Oct 30 21:47:05.964832 ip-10-0-135-59 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:47:05.964845 ip-10-0-135-59 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:47:05.964855 ip-10-0-135-59 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:47:05.965183 ip-10-0-135-59 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:47:17.256621 ip-10-0-135-59 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:47:17.256640 ip-10-0-135-59 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 572fbd6affdd4c21b61c0d3bdf6a4938 --
Oct 30 21:49:16.840938 ip-10-0-135-59 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:49:17.251675 ip-10-0-135-59 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:17.251675 ip-10-0-135-59 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:49:17.251675 ip-10-0-135-59 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:17.251675 ip-10-0-135-59 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:17.257115 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257037    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.257115 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257116    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257120    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257123    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257126    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257129    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257132    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257134    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257137    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257139    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257142    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257144    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257147    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257149    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257152    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257155    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257157    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257160    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257162    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257165    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257167    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.257184 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257170    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257178    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257181    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257183    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257186    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257189    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257192    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257195    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257197    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257200    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257202    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257205    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257208    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257210    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257213    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257215    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257217    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257220    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257222    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257225    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.257640 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257227    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257230    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257232    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257234    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257237    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257240    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257242    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257245    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257248    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257251    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257253    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257256    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257258    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257260    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257264    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257266    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257269    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257271    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257274    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257277    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.258151 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257279    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257281    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257284    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257286    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257289    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257292    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257294    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257297    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257299    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257301    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257304    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257306    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257309    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257311    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257314    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257316    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257318    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257321    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257324    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.258621 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257327    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257331    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257335    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257339    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257343    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257346    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257348    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257351    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257353    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257356    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257359    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257361    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257363    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257366    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257368    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257371    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257373    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257376    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257378    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.259097 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257380    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257383    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257385    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257387    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257390    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257392    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.257396    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258167    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258174    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258177    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258181    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258185    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258188    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258190    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258193    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258196    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258199    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258201    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258203    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258206    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.259549 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258208    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258211    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258214    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258217    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258219    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258222    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258224    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258227    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258229    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258232    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258234    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258237    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258239    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258241    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258244    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258246    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258249    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258251    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258253    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.260040 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258256    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258258    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258260    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258263    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258266    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258268    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258271    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258274    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258276    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258279    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258281    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258284    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258286    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258288    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258292    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258296    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258300    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258303    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258306    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.260475 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258310    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258313    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258315    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258318    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258320    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258323    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258325    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258328    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258330    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258332    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258335    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258337    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258340    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258342    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258345    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258347    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258350    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258352    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258355    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258359    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.260933 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258362    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258365    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258367    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258370    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258373    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258377    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258380    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258383    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258385    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258388    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258391    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258394    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258396    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258399    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258402    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258404    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258407    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258409    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258412    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.261394 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258414    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258417    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258419    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258422    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258424    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258427    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258429    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258431    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258434    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258436    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258439    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258441    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258446    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258448    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258451    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258454    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258921    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258927    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258930    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.261878 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258933    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258936    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258938    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258941    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258944    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258947    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258950    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258952    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258955    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258957    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258960    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258962    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258965    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258967    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258970    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258973    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258976    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258978    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258989    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.262323 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258992    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258994    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.258997    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259000    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259003    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259005    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259009    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259013    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259015    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259017    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259020    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259023    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259025    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259029    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259032    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259035    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259037    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259040    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259043    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259046    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.262790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259048    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259051    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259053    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259055    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259058    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259060    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259063    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259065    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259068    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259070    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259072    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259075    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259077    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259080    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259082    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259084    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259087    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259089    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259093    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259095    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.263297 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259097    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259100    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259104    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259107    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259110    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259112    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259114    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259118    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259120    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259123    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259125    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259128    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259131    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259133    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259135    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259138    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259140    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259142    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259145    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.263790 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259147    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259150    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259152    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259154    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259157    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259159    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259162    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259164    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259167    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259169    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259172    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259175    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259178    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259180    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259182    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259186    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259189    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259191    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259194    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259196    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.264230 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259199    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259202    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259204    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259206    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.259209    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261046    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261055    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261061    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261065    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261070    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261073    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261077    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261081    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261085    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261088    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261092    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261095    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261098    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261102    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261105    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261108    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261111    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261114    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:49:17.264773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261119    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261121    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261125    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261128    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261132    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261136    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261138    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261141    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261145    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261148    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261150    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261153    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261156    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261159    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261163    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261166    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261169    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261172    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261175    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261177    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261181    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261184    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261187    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261190    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:49:17.265293 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261193    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261197    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261200    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261202    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261205    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261208    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261211    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261214    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261217    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261220    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261223    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261226    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261230    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261233    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261236    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261239    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261242    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261245    2573 flags.go:64] FLAG: --help="false"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261248    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261251    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261255    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261257    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261260    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:49:17.265897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261264    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261267    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261270    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261272    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261275    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261278    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261281    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261284    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261286    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261289    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261292    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261295    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261298    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261301    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261303    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261306    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261312    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261315    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261318    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261320    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261323    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261326    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261329    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261332    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261336    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:49:17.266431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261339    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261343    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261346    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261348    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261351    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261354    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261357    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261360    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261362    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261377    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261380    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261383    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261386    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261388    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261392    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261394    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261397    2573 flags.go:64] FLAG: --port="10250"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261401    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261403    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01038558fee73058d"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261407    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261410    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261413    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261415    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261426    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261428    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261431    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:49:17.267044 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261434    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261437    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261440    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261443    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261446    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261449    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261451    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261454    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261457    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261460    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261463    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261465    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261468    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261471    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261474    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261477    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261480    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261483    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261486    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261489    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261492    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261497    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261500    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261504    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261512    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:49:17.267643 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261516    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261518    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261522    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261525    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261527    2573 flags.go:64] FLAG: --v="2"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261532    2573 flags.go:64] FLAG: --version="false"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261535    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261539    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261542    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261679    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261683    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261686    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261689    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261692    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261694    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261697    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261700    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261702    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261705    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261708    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261710    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.268577 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261713    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261715    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261718    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261721    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261723    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261726    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261728    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261730    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261733    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261737    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261740    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261744    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261746    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261749    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261751    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261754    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261756    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261758    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261761    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.269217 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261763    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261766    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261768    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261771    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261773    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261776    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261778    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261781    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261783    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261786    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261788    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261791    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261793    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261796    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261798    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261800    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261803    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261806    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261808    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.269686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261812    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261816    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261820    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261824    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261827    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261830    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261833    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261835    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261838    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261840    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261843    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261845    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261847    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261850    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261853    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261855    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261858    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261860    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261862    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.270188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261865    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261867    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261870    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261872    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261874    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261877    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261880    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261882    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261885    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261887    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261891    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261895    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261898    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261900    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261903    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261905    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261909    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261911    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261915    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261918    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.270634 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261920    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261923    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261925    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261928    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261930    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261933    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261935    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261938    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261940    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261943    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261945    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261948    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261950    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261953    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261955    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261958    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.261960    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.271188 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.261986    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262020    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262024    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262027    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262029    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262032    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262034    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262037    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262040    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262043    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262045    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262048    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262051    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262056    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262058    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262061    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262063    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262066    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262068    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262070    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.271588 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262073    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262075    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262078    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262080    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262083    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262085    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262088    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262090    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262092    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262095    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262098    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262100    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262102    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262105    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262107    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262109    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262112    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262114    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262117    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.272081 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262119    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262121    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262124    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262127    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262129    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262133    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262136    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262139    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262141    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262144    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262146    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262149    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262151    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262154    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262157    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262159    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262162    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262164    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262167    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.272542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262169    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262172    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262174    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262177    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262180    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262182    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262184    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262188    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262190    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262193    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262197    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262200    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262204    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262206    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262209    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262211    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262214    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262217    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262220    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.273011 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262227    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262229    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262232    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262234    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262237    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262239    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262241    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262244    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262247    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262249    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262252    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262254    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262257    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262259    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262261    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262264    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262266    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262269    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262271    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.273447 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262274    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262278    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262281    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262284    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262286    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262289    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262291    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262294    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262297    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262299    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.262301    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.270310    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.270328    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.271054    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.271061    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271124    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271129    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271132    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271135    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271138    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.274001 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271140    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271143    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271146    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271148    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271151    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271153    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271156    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271158    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271160    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271163    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271166    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271169    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271172    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271175    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271178    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271180    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271183    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271185    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271188    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.274472 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271191    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271194    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271197    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271199    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271202    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271205    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271208    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271211    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271214    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271216    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271219    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271221    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271224    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271226    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271229    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271232    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271234    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271237    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271239    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271241    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.274978 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271244    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271246    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271249    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271252    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271254    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271257    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271259    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271261    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271264    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271267    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271269    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271272    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271274    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271277    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271281    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271286    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271288    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271293    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.275449 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271297    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271300    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271303    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271306    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271308    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271311    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271313    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271316    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271318    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271321    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271323    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271326    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271329    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271331    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271333    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271336    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271338    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271341    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271344    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271347    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.275987 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271349    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271351    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271354    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271357    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271359    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271362    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271364    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271367    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271370    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271372    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271374    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271377    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271380    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271383    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271385    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271388    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271390    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271393    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271395    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271398    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.276542 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271400    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271403    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271405    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271407    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.271435    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271575    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271580    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271583    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271586    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271588    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271591    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271594    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271597    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271600    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271602    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.277033 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271605    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271608    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271611    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271613    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271616    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271618    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271621    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271623    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271626    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271628    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271631    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271633    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271636    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271639    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271641    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271644    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271647    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271649    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271651    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.277469 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271654    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271656    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271675    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271679    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271681    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271684    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271686    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271690    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271694    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271698    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271701    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271704    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271706    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271709    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271711    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271714    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271716    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271719    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271721    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.277938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271724    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271726    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271729    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271732    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271734    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271737    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271740    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271742    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271745    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271747    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271749    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271752    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271755    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271757    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271759    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271762    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271764    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271767    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271769    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271772    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.278454 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271774    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271777    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271779    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271783    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271785    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271787    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271790    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271792    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271795    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271797    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271800    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271802    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271804    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271807    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271809    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271812    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271815    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271817    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271819    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.278991 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271821    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271824    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271827    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271831    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271834    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271837    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271839    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271842    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271844    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271846    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271849    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271851    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271853    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271856    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271858    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271861    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271863    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271866    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.279767 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271868    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.271891    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271923    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271927    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271930    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271933    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271936    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271939    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271941    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271944    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271946    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271949    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271952    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271955    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271958    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:17.280326 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271961    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271964    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271966    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271968    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271971    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271973    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271976    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271978    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271980    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271983    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271985    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271988    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271990    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271992    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271995    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271997    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.271999    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272003    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272005    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272008    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:17.280689 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272010    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272013    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272015    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272017    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272020    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272022    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272025    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272027    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272030    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272032    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272035    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272037    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272039    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272042    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272044    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272046    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272049    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272051    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272054    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:17.281402 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272056    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272059    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272061    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272064    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272066    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272068    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272073    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272076    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272078    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272081    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272084    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272088    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272092    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272095    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272098    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272101    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272105    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272107    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272110    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:17.281938 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272112    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272115    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272117    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272120    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272122    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272126    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272128    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272131    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272133    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272136    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272138    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272141    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272143    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272146    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272148    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272150    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272153    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272155    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272158    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272161    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:17.282511 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272164    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272167    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272169    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272172    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272174    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272177    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272180    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272183    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272185    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272188    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272190    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272193    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272195    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272198    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:17.272200    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.272258    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.274534    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.275524    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.275644    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:17.283144 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.276373    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:17.283583 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.277692    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:17.283583 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.278763    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:17.289296 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.289281    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:49:17.293132 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.293120    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 21:49:17.293932 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.293909    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:49:17.298152 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.298134    2573 fs.go:135] Filesystem UUIDs: map[5558aabc-ed6a-4cfa-8117-61ded741325b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ec18faa8-19dc-46dd-914b-f0ac3e23e906:/dev/nvme0n1p3]
Oct 30 21:49:17.298197 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.298153    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:49:17.306509 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.306390    2573 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:17.304652288 +0000 UTC m=+0.350121757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101210 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec3f90aec1864ba74af74050015f5 SystemUUID:ec2ec3f9-0aec-1864-ba74-af74050015f5 BootID:572fbd6a-ffdd-4c21-b61c-0d3bdf6a4938 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:86:ec:78:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:86:ec:78:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:09:83:23:ff:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:49:17.306509 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.306502    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:49:17.306636 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.306608    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:49:17.310069 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310044    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:49:17.310197 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310072    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-59.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:49:17.310913 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310903    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:49:17.310943 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310915    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:49:17.310969 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310942    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:17.310969 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.310956    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:17.312164 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.312154    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:49:17.316435 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.316423    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:49:17.316539 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.316530    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:49:17.317345 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317336    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:49:17.317375 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317348    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:49:17.317375 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317358    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:49:17.317375 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317367    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:49:17.317375 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317374    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:49:17.317700 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.317688    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:17.319964 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.319943    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:49:17.326410 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.326398    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:17.327337 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327325    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:49:17.327376 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327362    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:49:17.327511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327496    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:49:17.327511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327512    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:49:17.327620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327518    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:49:17.327620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327523    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:49:17.327620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327528    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:49:17.327620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.327534    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:49:17.328275 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328267    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:49:17.328304 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328276    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:49:17.328304 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328284    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:49:17.328304 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328290    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:49:17.328304 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328298    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:49:17.328418 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.328307    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:49:17.329913 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.329904    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:49:17.329945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.329914    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:49:17.333591 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.333579    2573 server.go:1263] "Started kubelet"
Oct 30 21:49:17.333685 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.333637    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:49:17.333751 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.333709    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:49:17.333800 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.333770    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:49:17.334391 ip-10-0-135-59 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:49:17.335294 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.335277    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:49:17.336089 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.336076    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:49:17.339908 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.339880    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:49:17.342133 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.342117    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:17.342606 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.342592    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:49:17.342811 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.342771    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found"
Oct 30 21:49:17.343425 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343407    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:49:17.343480 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343408    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:49:17.343480 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343450    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:49:17.343542 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343506    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:49:17.343542 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343515    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:49:17.343542 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343525    2573 factory.go:55] Registering systemd factory
Oct 30 21:49:17.343542 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343537    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:49:17.343773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343756    2573 factory.go:153] Registering CRI-O factory
Oct 30 21:49:17.343773 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343775    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:49:17.343922 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343834    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:49:17.343922 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343854    2573 factory.go:103] Registering Raw factory
Oct 30 21:49:17.343922 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.343865    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:49:17.344890 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.344877    2573 manager.go:319] Starting recovery of all containers
Oct 30 21:49:17.357167 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.357148    2573 manager.go:324] Recovery completed
Oct 30 21:49:17.361016 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.361004    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.362492 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.362474    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.362553 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.362506    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.362553 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.362516    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.363008 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.362992    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:49:17.363058 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.363010    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:49:17.363058 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.363028    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:49:17.363128 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.362996    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.363777 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.363749    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.363839 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.363781    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.363839 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.363795    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.365362 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.365348    2573 policy_none.go:50] "Start"
Oct 30 21:49:17.365398 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.365369    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:49:17.365398 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.365382    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:49:17.367534 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.367522    2573 policy_none.go:44] "Start"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.407249    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.411836    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.412517    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.412532    2573 server.go:87] "Starting device plugin registration server"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.412778    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.412791    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.412897    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.413000    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.413007    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.413410    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.413462    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-59.us-west-1.compute.internal\" not found"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.417821    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.419111    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.419130    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.419148    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:49:17.430443 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.419180    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:49:17.513028 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.512974    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.513805 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.513788    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.513903 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.513823    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.513903 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.513836    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.513903 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.513874    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.520275 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.520258    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"]
Oct 30 21:49:17.520345 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.520308    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.521037 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.521019    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.521121 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.521049    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.521121 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.521060    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.523034 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523022    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.523698 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523679    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.523774 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523711    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.523774 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523723    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.523938 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.523974 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.523954    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.524615 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.524602    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.524696 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.524630    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.524696 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.524640    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.525491 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.525479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.525546 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.525501    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.526123 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.526109    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.526215 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.526133    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.526215 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.526147    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.528224 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.528205    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-59.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:49:17.528330 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.528248    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-59.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:49:17.529100 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.528205    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-59.us-west-1.compute.internal.18e36d07e722017c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-59.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-59.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:17.333553532 +0000 UTC m=+0.379022998,LastTimestamp:2026-10-30 21:49:17.333553532 +0000 UTC m=+0.379022998,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-59.us-west-1.compute.internal,}"
Oct 30 21:49:17.553637 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.553616    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.556328 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.556310    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.566543 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.566525    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:17.566624 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.566572    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gtbw5"
Oct 30 21:49:17.589712 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.589615    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-59.us-west-1.compute.internal.18e36d07e8db9081  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-59.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-59.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-59.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:17.362491521 +0000 UTC m=+0.407960992,LastTimestamp:2026-10-30 21:49:17.362491521 +0000 UTC m=+0.407960992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-59.us-west-1.compute.internal,}"
Oct 30 21:49:17.589819 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.589704    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:49:17.589819 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.589723    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:49:17.589915 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.589854    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 21:49:17.644608 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.644580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.644705 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.644608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.644705 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.644628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.649280 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.649264    2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.649410 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.649396    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:17.745514 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.745514 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.745682 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.745682 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745593    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.745682 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745596    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.745682 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.745604    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.834203 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:17.834141    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.849611 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.849592    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:17.850440 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.850424    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:17.850493 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.850457    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:17.850493 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.850472    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:17.850549 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.850497    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.857593 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.857580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.862624 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.862571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.908285 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.908255    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:17.942825 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:17.942802    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:18.005612 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.005593    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:18.006870 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.006850    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:18.066923 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.066895    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:18.274897 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.274816    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:49:18.275444 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.274953    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa7caa7bfd2874950af3c2a870e3e248-6e00b72b7f2ae867.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-59.us-west-1.compute.internal.18e36d07e8db9081\": read tcp 10.0.135.59:50188->18.227.6.60:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-59.us-west-1.compute.internal.18e36d07e8db9081  default   5078 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-59.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-59.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-59.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:17 +0000 UTC,LastTimestamp:2026-10-30 21:49:17.521036345 +0000 UTC m=+0.566505817,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-59.us-west-1.compute.internal,}"
Oct 30 21:49:18.318027 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.318000    2573 apiserver.go:52] "Watching apiserver"
Oct 30 21:49:18.342829 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.342805    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:18.572933 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.572857    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:17 +0000 UTC" deadline="2028-03-25 01:35:19.744328104 +0000 UTC"
Oct 30 21:49:18.572933 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.572883    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12267h46m1.171447368s"
Oct 30 21:49:18.596611 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.596583    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:49:18.596926 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.596908    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-ksctb","openshift-multus/network-metrics-daemon-j9lzw","openshift-network-diagnostics/network-check-target-6qwzk","openshift-network-operator/iptables-alerter-md9xp","openshift-ovn-kubernetes/ovnkube-node-wq75f","kube-system/global-pull-secret-syncer-jhchm","kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d","openshift-multus/multus-additional-cni-plugins-hxvm4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal","kube-system/konnectivity-agent-nj82c","openshift-dns/node-resolver-t8f8p","openshift-image-registry/node-ca-zpqg6","openshift-multus/multus-2vmrb"]
Oct 30 21:49:18.599162 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.599146    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.601122 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.601104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.601203 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.601167    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:18.602985 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.602960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:18.603079 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.603018    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:18.604899 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.604884    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.606913 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.606897    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.609020 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.609005    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.609084 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.609050    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:18.611025 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.611013    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.613187 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.613169    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.615342 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.615325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.617480 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.617464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.619616 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.619602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.621844 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.621827    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.644733 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.644712    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:49:18.650537 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.650629 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650545    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.650629 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.650629 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.650757 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.650757 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.650824 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.650824 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.650899 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.650899 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.650964 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.650964 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651043 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650966    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.651043 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.650989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7z5\" (UniqueName: \"kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651043 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbl6\" (UniqueName: \"kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.651043 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651043 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.651234 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651253    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651288    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfpt\" (UniqueName: \"kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.651585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs59\" (UniqueName: \"kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b22\" (UniqueName: \"kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651727    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn95r\" (UniqueName: \"kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.652004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651968    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.651991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652035    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c5r\" (UniqueName: \"kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qs7w\" (UniqueName: \"kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652511 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652419    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzl4\" (UniqueName: \"kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.652945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.652477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.654517 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.654501    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.654611 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.654596    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\""
Oct 30 21:49:18.654759 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.654746    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.658609 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.658586    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\""
Oct 30 21:49:18.659562 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.659545    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.659638 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.659565    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.659638 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.659618    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:49:18.661435 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.661421    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:49:18.661886 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.661874    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.662345 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.662330    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:49:18.662650 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.662638    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.662733 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.662704    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:49:18.662784 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.662771    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\""
Oct 30 21:49:18.662832 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.662820    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:49:18.664984 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.664968    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\""
Oct 30 21:49:18.665035 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.664987    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:49:18.701344 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701325    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.701461 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701375    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.701461 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701391    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.701461 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701391    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.701461 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701383    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:49:18.701461 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.701403    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:49:18.704004 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.703975    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:49:18.704182 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.704168    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\""
Oct 30 21:49:18.704945 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.704930    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\""
Oct 30 21:49:18.732482 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.732468    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:18.752048 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752029    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:18.752489 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752474    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:18.752562 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752548    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tbw8r"
Oct 30 21:49:18.752649 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.752705 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752657    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.752743 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.752743 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.752803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752740    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.752803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.752803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.752803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.752803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752797    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzl4\" (UniqueName: \"kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752871    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752933    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752942    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752940    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752951    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.752965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753030 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753026    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753110    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753232    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753271    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753273    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753356    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753368    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753409    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:49:18.753710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753369    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7z5\" (UniqueName: \"kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbl6\" (UniqueName: \"kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753613    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753718    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.754400 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753872    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753923    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753935    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753966    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.753925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754208    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfpt\" (UniqueName: \"kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.755395 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs59\" (UniqueName: \"kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b22\" (UniqueName: \"kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn95r\" (UniqueName: \"kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754452    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754492    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754558    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\""
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754582    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\""
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754613    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\""
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754687    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.756155 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754726    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754458    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754827    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754863    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754875    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c5r\" (UniqueName: \"kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754944    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.755015    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:18.757036 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.754908    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755433    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755544    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755548    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755584    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.755609    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.755696    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.255650287 +0000 UTC m=+2.301119768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs7w\" (UniqueName: \"kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.755945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.756036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.757710 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.756307    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.757621    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.757764    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.757784    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.757845    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.757856    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758093    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758274    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.758373 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758330    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.758714 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758485    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.758714 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758573    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:49:18.758714 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758592    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:18.758846 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758729    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.758846 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758732    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.758846 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758798    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.758988 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.758920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.759038 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.759028    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.259006823 +0000 UTC m=+2.304476292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:18.759038 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759027    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759263    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759357    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759375    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759453    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759574    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759575    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759616    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759626    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759632    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.759694 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759693    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.760347 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.759831    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.760347 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.760036    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.760347 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.760089    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.760347 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.760207    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:49:18.760347 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.760220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.761008 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.760984    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.761120 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.761100    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.762332 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.762309    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.763236 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.763220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.763322 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.763306    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:18.822360 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822332    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b22\" (UniqueName: \"kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:18.822487 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822470    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzl4\" (UniqueName: \"kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:18.822546 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822516    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c5r\" (UniqueName: \"kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:18.822603 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822562    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbl6\" (UniqueName: \"kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.822894 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822870    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs59\" (UniqueName: \"kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:18.823008 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.822874    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7z5\" (UniqueName: \"kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.823150 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.823134    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfpt\" (UniqueName: \"kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:18.824178 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.824156    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn95r\" (UniqueName: \"kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:18.824178 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.824174    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs7w\" (UniqueName: \"kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.824280 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.824228    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:18.824280 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.824245    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:18.824280 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.824254    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:18.824398 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:18.824317    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.324302406 +0000 UTC m=+2.369771873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:18.883039 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.883014    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:18.929084 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.929053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:18.947857 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.947832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:18.978988 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.978964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:18.983424 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:18.983409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:19.003412 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.003391    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:19.009218 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.009197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:19.017694 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.017655    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c256e02c33d90178c37b95ddf27e766.slice/crio-3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f WatchSource:0}: Error finding container 3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f: Status 404 returned error can't find the container with id 3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f
Oct 30 21:49:19.018459 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.018444    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106668d41ee4901fe4ed1e1a301dc3c3.slice/crio-7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e WatchSource:0}: Error finding container 7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e: Status 404 returned error can't find the container with id 7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e
Oct 30 21:49:19.019631 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.019613    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f5e49_2f17_4384_b1f4_32521160dd20.slice/crio-293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea WatchSource:0}: Error finding container 293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea: Status 404 returned error can't find the container with id 293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea
Oct 30 21:49:19.020178 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.020154    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adc69d5_6bad_412c_abe9_687f28a7a590.slice/crio-540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934 WatchSource:0}: Error finding container 540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934: Status 404 returned error can't find the container with id 540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934
Oct 30 21:49:19.022073 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.022052    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd8c8b1_9d40_4427_9a3c_71a449e6af59.slice/crio-cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448 WatchSource:0}: Error finding container cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448: Status 404 returned error can't find the container with id cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448
Oct 30 21:49:19.023422 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.023319    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:19.024058 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.024043    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f8a528_1c9b_4f51_8e8f_f298e8982c98.slice/crio-6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5 WatchSource:0}: Error finding container 6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5: Status 404 returned error can't find the container with id 6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5
Oct 30 21:49:19.028289 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.028270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:19.034288 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.034271    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:19.037729 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.037700    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1bf1e8_1b8d_4781_ae6f_ac2010aad2d9.slice/crio-14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258 WatchSource:0}: Error finding container 14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258: Status 404 returned error can't find the container with id 14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258
Oct 30 21:49:19.037803 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.037763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:19.042611 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.042592    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaa0470_d397_4d38_b674_ff07fee73459.slice/crio-62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22 WatchSource:0}: Error finding container 62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22: Status 404 returned error can't find the container with id 62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22
Oct 30 21:49:19.046570 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.046552    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0294bcd1_88a6_47de_a415_61451b91c9c2.slice/crio-6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd WatchSource:0}: Error finding container 6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd: Status 404 returned error can't find the container with id 6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd
Oct 30 21:49:19.261057 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.261035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:19.261216 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.261066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:19.261216 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.261147    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:19.261216 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.261169    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:19.261216 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.261204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.261177775 +0000 UTC m=+3.306647228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:19.261364 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.261226    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.261220083 +0000 UTC m=+3.306689536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:19.361578 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.361549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:19.362116 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.361688    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:19.362116 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.361706    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:19.362116 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.361716    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:19.362116 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:19.361764    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.361749029 +0000 UTC m=+3.407218481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:19.423432 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.423391    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2vmrb" event={"ID":"0294bcd1-88a6-47de-a415-61451b91c9c2","Type":"ContainerStarted","Data":"6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd"}
Oct 30 21:49:19.424304 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.424285    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8f8p" event={"ID":"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9","Type":"ContainerStarted","Data":"14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258"}
Oct 30 21:49:19.425168 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.425152    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nj82c" event={"ID":"72f8a528-1c9b-4f51-8e8f-f298e8982c98","Type":"ContainerStarted","Data":"6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5"}
Oct 30 21:49:19.426023 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.426008    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerStarted","Data":"cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448"}
Oct 30 21:49:19.426829 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.426802    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d" event={"ID":"3adc69d5-6bad-412c-abe9-687f28a7a590","Type":"ContainerStarted","Data":"540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934"}
Oct 30 21:49:19.427598 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.427583    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerStarted","Data":"3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f"}
Oct 30 21:49:19.428447 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.428432    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zpqg6" event={"ID":"acaa0470-d397-4d38-b674-ff07fee73459","Type":"ContainerStarted","Data":"62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22"}
Oct 30 21:49:19.429261 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.429246    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea"}
Oct 30 21:49:19.430084 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.430068    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"106668d41ee4901fe4ed1e1a301dc3c3","Type":"ContainerStarted","Data":"7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e"}
Oct 30 21:49:19.645497 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.645475    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d37bdc1_5888_4d01_bd66_90f17ef65d8e.slice/crio-2a8d68b7d0aea4cb1e1cec61a4dc448ada6a585d59d0319251a4273b0346fbaa WatchSource:0}: Error finding container 2a8d68b7d0aea4cb1e1cec61a4dc448ada6a585d59d0319251a4273b0346fbaa: Status 404 returned error can't find the container with id 2a8d68b7d0aea4cb1e1cec61a4dc448ada6a585d59d0319251a4273b0346fbaa
Oct 30 21:49:19.702686 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:19.702644    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e83dd_993e_4e5c_ab7f_ef4db6f57f05.slice/crio-108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e WatchSource:0}: Error finding container 108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e: Status 404 returned error can't find the container with id 108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e
Oct 30 21:49:19.762993 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.762963    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:18 +0000 UTC" deadline="2028-04-23 15:25:49.090280589 +0000 UTC"
Oct 30 21:49:19.762993 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:19.762987    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12977h36m29.327295971s"
Oct 30 21:49:20.268559 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.268523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:20.268765 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.268609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:20.268829 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.268773    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:20.268873 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.268827    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:22.26881008 +0000 UTC m=+5.314279534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:20.268956 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.268894    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:20.268956 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.268927    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:22.268916183 +0000 UTC m=+5.314385650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:20.369871 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.369836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:20.370307 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.370007    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:20.370307 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.370029    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:20.370307 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.370041    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:20.370307 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.370092    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:22.370076308 +0000 UTC m=+5.415545766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:20.419771 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.419739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:20.419923 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.419864    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:20.420000 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.419937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:20.420072 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.420007    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:20.420127 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.420074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:20.420176 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:20.420154    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:20.449692 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.446699    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-md9xp" event={"ID":"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05","Type":"ContainerStarted","Data":"108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e"}
Oct 30 21:49:20.459895 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.459862    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ksctb" event={"ID":"9d37bdc1-5888-4d01-bd66-90f17ef65d8e","Type":"ContainerStarted","Data":"2a8d68b7d0aea4cb1e1cec61a4dc448ada6a585d59d0319251a4273b0346fbaa"}
Oct 30 21:49:20.763457 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.763379    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:18 +0000 UTC" deadline="2028-06-16 06:28:08.635932281 +0000 UTC"
Oct 30 21:49:20.763457 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:20.763416    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14264h38m47.872519341s"
Oct 30 21:49:22.285269 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.285238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:22.285806 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.285284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:22.285806 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.285451    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:22.285806 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.285507    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.285495494 +0000 UTC m=+9.330964948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:22.285976 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.285897    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:22.285976 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.285946    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.285930557 +0000 UTC m=+9.331400016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:22.385801 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.385711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:22.385966 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.385885    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:22.385966 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.385902    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:22.385966 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.385914    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:22.385966 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.385960    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.385947653 +0000 UTC m=+9.431417108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:22.419794 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.419764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:22.419956 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.419889    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:22.420017 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.419966    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:22.420213 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.420077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:22.420213 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:22.420194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:22.420353 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:22.420287    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:23.488916 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:23.488703    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="a947e9e465c9d61d1b89a867565c0fdbe53a32598e2763cd9a8452e5bb8f8dd8" exitCode=0
Oct 30 21:49:23.489324 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:23.488761    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"a947e9e465c9d61d1b89a867565c0fdbe53a32598e2763cd9a8452e5bb8f8dd8"}
Oct 30 21:49:24.419693 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.419602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:24.419890 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:24.419749    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:24.420133 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.420105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:24.420260 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:24.420237    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:24.420335 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.420296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:24.420432 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:24.420391    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:24.494623 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.494584    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nj82c" event={"ID":"72f8a528-1c9b-4f51-8e8f-f298e8982c98","Type":"ContainerStarted","Data":"5670d0564ec722d8b5aa579b053d7c906d8d3a101100b4b170287729e8819cda"}
Oct 30 21:49:24.501625 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.501595    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c256e02c33d90178c37b95ddf27e766" containerID="e785cf876a03872ab6ea0d59b570af456330f03b5eb624f4356782885bb75503" exitCode=0
Oct 30 21:49:24.501780 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.501699    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerDied","Data":"e785cf876a03872ab6ea0d59b570af456330f03b5eb624f4356782885bb75503"}
Oct 30 21:49:24.511654 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.511631    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"106668d41ee4901fe4ed1e1a301dc3c3","Type":"ContainerStarted","Data":"9533cf65f618d712a42b04087ce7535c5decdbf02ea90fe3fde052339c5f2632"}
Oct 30 21:49:24.736680 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.734092    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nj82c" podStartSLOduration=3.11163945 podStartE2EDuration="7.734078119s" totalImagesPullingTime="4.622438669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.615479922 +0000 UTC m=+7.660949401" watchObservedRunningTime="2026-10-30 21:49:24.734078119 +0000 UTC m=+7.779547622"
Oct 30 21:49:24.854537 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:24.854478    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal" podStartSLOduration=7.854456743 podStartE2EDuration="7.854456743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.853789414 +0000 UTC m=+7.899258891" watchObservedRunningTime="2026-10-30 21:49:24.854456743 +0000 UTC m=+7.899926223"
Oct 30 21:49:25.523620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.523584    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerStarted","Data":"d8943ba3d1a08a8793905bbc31b4fa3dcfa36197cb4e3b7cc48dada30f7e23d4"}
Oct 30 21:49:25.530276 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.530247    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zpqg6" event={"ID":"acaa0470-d397-4d38-b674-ff07fee73459","Type":"ContainerStarted","Data":"1eb6463d0d06faaa76ce64511d54beb46c6fbeaf2818ba4e519734a989b9a459"}
Oct 30 21:49:25.645635 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.645572    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" podStartSLOduration=8.645554467 podStartE2EDuration="8.645554467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.645340706 +0000 UTC m=+8.690810185" watchObservedRunningTime="2026-10-30 21:49:25.645554467 +0000 UTC m=+8.691023945"
Oct 30 21:49:25.700028 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.699997    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:25.700772 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.700750    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:25.764297 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:25.764241    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zpqg6" podStartSLOduration=2.88707751 podStartE2EDuration="8.764223102s" totalImagesPullingTime="5.877145592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.763452967 +0000 UTC m=+8.808922444" watchObservedRunningTime="2026-10-30 21:49:25.764223102 +0000 UTC m=+8.809692579"
Oct 30 21:49:26.320440 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.320353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:26.320589 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.320461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:26.320589 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.320575    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:26.320718 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.320632    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:34.320613335 +0000 UTC m=+17.366082802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:26.321117 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.321025    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:26.321117 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.321079    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:34.321064766 +0000 UTC m=+17.366534220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:26.420138 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.420090    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:26.420309 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420214    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:26.420309 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.420280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:26.420416 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:26.420416 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.420405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:26.420513 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420477    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:26.420792 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.420768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:26.420931 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420885    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:26.420931 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420906    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:26.420931 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420919    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:26.421086 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:26.420968    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:34.420950645 +0000 UTC m=+17.466420115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:26.539024 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:26.538989    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d" event={"ID":"3adc69d5-6bad-412c-abe9-687f28a7a590","Type":"ContainerStarted","Data":"a1c11fc6751fe4df564001ac24833605f08a86042eedb419fa8baf4d41b542cc"}
Oct 30 21:49:27.522852 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:27.522639    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:49:27.543790 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:27.543772    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:27.544256 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:27.544228    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d" event={"ID":"3adc69d5-6bad-412c-abe9-687f28a7a590","Type":"ContainerStarted","Data":"f8f676641e05105ba561bdb96dc3db58e4456b4393483c52b5cfb8a4058ab5e6"}
Oct 30 21:49:28.419953 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.419917    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:28.420092 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:28.420045    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:28.420158 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.420135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:28.420237 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:28.420218    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:28.420281 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.420274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:28.420352 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:28.420334    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:28.426825 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.426719    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:27.52265393Z","UUID":"3a826195-2524-4414-8aaf-80c577cf41ec","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:49:28.428997 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.428946    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:49:28.429115 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.428975    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:49:28.549377 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.549346    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2vmrb" event={"ID":"0294bcd1-88a6-47de-a415-61451b91c9c2","Type":"ContainerStarted","Data":"0f5ea7cc431a7f10573ce56aa1669c0d67ebc534fc5f0383ecd5a1787d0d2807"}
Oct 30 21:49:28.552940 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.552915    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8f8p" event={"ID":"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9","Type":"ContainerStarted","Data":"2be0bfbea9aa089bb538db33a27649a188a8f80f534bbed5eeeb8d0c89986868"}
Oct 30 21:49:28.555521 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.555488    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="987527a9780397cf08383ca751a35ad54008456334ffb1023817421127fead9c" exitCode=0
Oct 30 21:49:28.555521 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.555521    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"987527a9780397cf08383ca751a35ad54008456334ffb1023817421127fead9c"}
Oct 30 21:49:28.804111 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.803797    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2vmrb" podStartSLOduration=3.303987564 podStartE2EDuration="11.803776489s" totalImagesPullingTime="8.499788925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:28.676797333 +0000 UTC m=+11.722266808" watchObservedRunningTime="2026-10-30 21:49:28.803776489 +0000 UTC m=+11.849245967"
Oct 30 21:49:28.922849 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:28.922790    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8f8p" podStartSLOduration=3.46053293 podStartE2EDuration="11.922769768s" totalImagesPullingTime="8.462236838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:28.922079684 +0000 UTC m=+11.967549160" watchObservedRunningTime="2026-10-30 21:49:28.922769768 +0000 UTC m=+11.968239246"
Oct 30 21:49:29.561851 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:29.561809    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d" event={"ID":"3adc69d5-6bad-412c-abe9-687f28a7a590","Type":"ContainerStarted","Data":"ce3ce18b7246b39c8983087d850d525a35944bcd270a283e6096a25f62e157ae"}
Oct 30 21:49:29.686585 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:29.685956    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d" podStartSLOduration=3.270257996 podStartE2EDuration="12.685934991s" totalImagesPullingTime="9.415676995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:29.685107411 +0000 UTC m=+12.730576888" watchObservedRunningTime="2026-10-30 21:49:29.685934991 +0000 UTC m=+12.731404469"
Oct 30 21:49:30.419635 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.419561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:30.419635 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.419578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:30.419852 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:30.419755    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:30.419852 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.419790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:30.419954 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:30.419861    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:30.419954 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:30.419907    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:30.566807 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.566772    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="11bef436afe22c212c32b04498c395c1ff2044b4579d232a519d40f8539d8294" exitCode=0
Oct 30 21:49:30.567388 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.566870    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"11bef436afe22c212c32b04498c395c1ff2044b4579d232a519d40f8539d8294"}
Oct 30 21:49:30.568515 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.568489    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-md9xp" event={"ID":"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05","Type":"ContainerStarted","Data":"8f206652f17e1f953b6498d1ef66a1ed090a73d1175970c9a6ab23cc309750d1"}
Oct 30 21:49:30.570216 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.570192    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ksctb" event={"ID":"9d37bdc1-5888-4d01-bd66-90f17ef65d8e","Type":"ContainerStarted","Data":"c4180dc05c78933b3b3e083c7eed3564d496b5b9610c8a528cad382cab9702bb"}
Oct 30 21:49:30.818296 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.818186    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ksctb" podStartSLOduration=3.358898708 podStartE2EDuration="13.818171098s" totalImagesPullingTime="10.45927239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.817785718 +0000 UTC m=+13.863255195" watchObservedRunningTime="2026-10-30 21:49:30.818171098 +0000 UTC m=+13.863640574"
Oct 30 21:49:30.936426 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:30.936364    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-md9xp" podStartSLOduration=5.569140367 podStartE2EDuration="13.936349727s" totalImagesPullingTime="8.36720936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.935973723 +0000 UTC m=+13.981443200" watchObservedRunningTime="2026-10-30 21:49:30.936349727 +0000 UTC m=+13.981819202"
Oct 30 21:49:32.420145 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:32.420095    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:32.420916 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:32.420104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:32.420916 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:32.420226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:32.420916 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:32.420230    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:32.420916 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:32.420333    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:32.420916 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:32.420407    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:32.577301 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:32.577267    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="16ecf9f64d3f729318a6fc4570a1bad98a318a2b5bd8e9cc9c65d7158e5ada3f" exitCode=0
Oct 30 21:49:32.577456 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:32.577314    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"16ecf9f64d3f729318a6fc4570a1bad98a318a2b5bd8e9cc9c65d7158e5ada3f"}
Oct 30 21:49:34.388892 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.388632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:34.389349 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.388957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:34.389349 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.389133    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:34.389349 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.389224    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret podName:8b478c4f-7211-4b12-ab86-cd866de1ae8a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.389206448 +0000 UTC m=+33.434675922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret") pod "global-pull-secret-syncer-jhchm" (UID: "8b478c4f-7211-4b12-ab86-cd866de1ae8a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:34.389349 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.389332    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:34.389528 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.389375    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.38936294 +0000 UTC m=+33.434832408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:34.419781 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.419753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:34.419781 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.419775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:34.419970 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.419899    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:34.419970 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.419930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:34.420055 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.419992    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:34.420055 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.420032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:34.489500 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.489473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:34.489639 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.489624    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:34.489718 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.489650    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:34.489718 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.489684    2573 projected.go:196] Error preparing data for projected volume kube-api-access-k7skz for pod openshift-network-diagnostics/network-check-target-6qwzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:34.489812 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:34.489751    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz podName:df135a70-eac2-457e-b470-bd45b99875c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.489731053 +0000 UTC m=+33.535200507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7skz" (UniqueName: "kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz") pod "network-check-target-6qwzk" (UID: "df135a70-eac2-457e-b470-bd45b99875c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:34.585439 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.585383    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"816c6c0e7c6a2b8bd2340e4dc4a6ff3758952ff63adc003e424e3e883250a634"}
Oct 30 21:49:34.585535 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.585467    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"440b4e6cad9c04ee639148d5aff70ac7c91bd8e2b9307bd3eec7b9d8828ab757"}
Oct 30 21:49:34.585535 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.585520    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"6cb9bb691f5d83041ccf232f454e769193fa5133caf6dd604b0847e2bcb4f322"}
Oct 30 21:49:34.585642 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.585565    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"56e3e544b88c8c83be57bbe815388ba9b1d265357672af3bc1c1891a30f06198"}
Oct 30 21:49:34.585642 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:34.585610    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"3725633b622eb1fcc72167adb00c224d57bdbf34dfc2e7f14bc56708b1652ef1"}
Oct 30 21:49:35.590431 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:35.590245    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="7af27c842272ef86b11dd8d27b360947590a8f3627f8df7853481ae3c0c6acd9" exitCode=0
Oct 30 21:49:35.590894 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:35.590316    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"7af27c842272ef86b11dd8d27b360947590a8f3627f8df7853481ae3c0c6acd9"}
Oct 30 21:49:35.593508 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:35.593487    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"a98dbfb2792e2f7aa9b76cf9ed2fa9351d4dfa7876b0ef18723a3f84de75d42c"}
Oct 30 21:49:36.345653 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.345628    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:36.345831 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.345747    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:36.346134 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.346121    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:36.419954 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.419929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:36.420103 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.420021    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:36.420103 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:36.420032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:36.420103 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.420050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:36.420224 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:36.420103    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:36.420224 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:36.420171    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:36.597863 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.597783    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="e398f4ee763eac94725169452822068274ca27a12789b23752b2270691f38afc" exitCode=0
Oct 30 21:49:36.598275 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:36.597867    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerDied","Data":"e398f4ee763eac94725169452822068274ca27a12789b23752b2270691f38afc"}
Oct 30 21:49:37.602750 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:37.602720    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" event={"ID":"3bd8c8b1-9d40-4427-9a3c-71a449e6af59","Type":"ContainerStarted","Data":"6ad2271580e90fdefe7cc06125a7a03aba2ff90f6f58f1e3629ae8f2a8ec0e68"}
Oct 30 21:49:37.605790 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:37.605768    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"4b9b9ae9efd0afdbeebd8dd721d367c49567c7cdef4c6b28ad58eb454ef9450f"}
Oct 30 21:49:37.736536 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:37.736492    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hxvm4" podStartSLOduration=8.371521788 podStartE2EDuration="20.736479303s" totalImagesPullingTime="12.364957515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.736255124 +0000 UTC m=+20.781724595" watchObservedRunningTime="2026-10-30 21:49:37.736479303 +0000 UTC m=+20.781948779"
Oct 30 21:49:38.420103 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:38.420070    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:38.420259 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:38.420149    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:38.420259 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:38.420230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:38.420259 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:38.420235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:38.420402 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:38.420320    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:38.420402 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:38.420367    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:39.613354 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:39.613166    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" event={"ID":"ea0f5e49-2f17-4384-b1f4-32521160dd20","Type":"ContainerStarted","Data":"bda2395592b10e93a53fa289a3ca4c94f29d4f39d9079ee2602781a94e524824"}
Oct 30 21:49:39.613989 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:39.613392    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:39.629285 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:39.629264    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:39.895344 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:39.895232    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f" podStartSLOduration=7.833763949 podStartE2EDuration="22.895217272s" totalImagesPullingTime="15.061453323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.869375704 +0000 UTC m=+22.914845180" watchObservedRunningTime="2026-10-30 21:49:39.895217272 +0000 UTC m=+22.940686751"
Oct 30 21:49:40.420302 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.420265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:40.420302 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.420292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:40.420547 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.420377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:40.420547 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:40.420383    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:40.420547 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:40.420444    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:40.420547 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:40.420511    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:40.614827 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.614798    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:40.615236 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.615220    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:40.630391 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:40.630369    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:41.535112 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.535040    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j9lzw"]
Oct 30 21:49:41.535286 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.535150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:41.535286 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:41.535232    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:41.538887 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.538853    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6qwzk"]
Oct 30 21:49:41.539021 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.538967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:41.539088 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:41.539066    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:41.539528 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.539509    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhchm"]
Oct 30 21:49:41.539620 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.539608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:41.539714 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:41.539699    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:41.616854 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:41.616830    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:42.619213 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:42.619187    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:43.419841 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:43.419808    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:43.420010 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:43.419815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:43.420010 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:43.419815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:43.420010 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:43.419932    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:43.420010 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:43.420003    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:43.420196 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:43.420081    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:45.420215 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:45.420184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:45.420215 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:45.420199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:45.420740 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:45.420286    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:45.420740 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:45.420340    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:45.420740 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:45.420417    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:45.420740 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:45.420533    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:47.420343 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:47.420314    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:47.420818 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:47.420391    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:47.420818 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:47.420438    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9lzw" podUID="1de1bbac-ecd0-4ac7-8357-71424716cb03"
Oct 30 21:49:47.420818 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:47.420452    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6qwzk" podUID="df135a70-eac2-457e-b470-bd45b99875c6"
Oct 30 21:49:47.420818 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:47.420489    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:47.420818 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:47.420554    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhchm" podUID="8b478c4f-7211-4b12-ab86-cd866de1ae8a"
Oct 30 21:49:47.704896 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:47.704832    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:49:47.705017 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:47.704942    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:49:48.318725 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:48.318685    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:48.318961 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:48.318948    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:48.337005 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:48.336978    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:49.420348 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.420313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:49.420348 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.420337    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:49.420839 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.420576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:49.475333 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.475306    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.475530 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.475513    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.525959 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.525931    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:49:49.526074 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.525967    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\""
Oct 30 21:49:49.526074 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.526053    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:49:49.526196 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.526183    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\""
Oct 30 21:49:49.526229 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.526199    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\""
Oct 30 21:49:49.716941 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:49.716880    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8f8p_0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9/dns-node-resolver/0.log"
Oct 30 21:49:50.400676 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.400625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:50.400676 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.400683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:50.400948 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:50.400804    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:49:50.400948 ip-10-0-135-59 kubenswrapper[2573]: E1030 21:49:50.400882    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs podName:1de1bbac-ecd0-4ac7-8357-71424716cb03 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:22.400862504 +0000 UTC m=+65.446331986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs") pod "network-metrics-daemon-j9lzw" (UID: "1de1bbac-ecd0-4ac7-8357-71424716cb03") : secret "metrics-daemon-secret" not found
Oct 30 21:49:50.404224 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.404207    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:50.449573 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.449547    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:50.501332 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.501305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:50.503787 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.503768    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:50.538676 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.538645    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zpqg6_acaa0470-d397-4d38-b674-ff07fee73459/node-ca/0.log"
Oct 30 21:49:50.702232 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.702162    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:50.896685 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:50.895027    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhchm"]
Oct 30 21:49:50.898274 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:50.898250    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b478c4f_7211_4b12_ab86_cd866de1ae8a.slice/crio-a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13 WatchSource:0}: Error finding container a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13: Status 404 returned error can't find the container with id a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13
Oct 30 21:49:51.079311 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:51.079237    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6qwzk"]
Oct 30 21:49:51.082148 ip-10-0-135-59 kubenswrapper[2573]: W1030 21:49:51.082113    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf135a70_eac2_457e_b470_bd45b99875c6.slice/crio-e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d WatchSource:0}: Error finding container e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d: Status 404 returned error can't find the container with id e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d
Oct 30 21:49:51.638723 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:51.638684    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6qwzk" event={"ID":"df135a70-eac2-457e-b470-bd45b99875c6","Type":"ContainerStarted","Data":"e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d"}
Oct 30 21:49:51.639793 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:51.639765    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jhchm" event={"ID":"8b478c4f-7211-4b12-ab86-cd866de1ae8a","Type":"ContainerStarted","Data":"a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13"}
Oct 30 21:49:55.650734 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:55.650697    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6qwzk" event={"ID":"df135a70-eac2-457e-b470-bd45b99875c6","Type":"ContainerStarted","Data":"f8dec8d46ff822fa5606b2284c70bad61c98aac568e5886203c2ccf70f09204c"}
Oct 30 21:49:55.651193 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:55.650817    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:55.769264 ip-10-0-135-59 kubenswrapper[2573]: I1030 21:49:55.769202    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6qwzk" podStartSLOduration=34.774551662 podStartE2EDuration="38.769182658s" totalImagesPullingTime="3.994630996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:55.76888028 +0000 UTC m=+38.814349758" watchObservedRunningTime="2026-10-30 21:49:55.769182658 +0000 UTC m=+38.814652136"
Oct 30 21:49:57.299182 ip-10-0-135-59 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:49:57.311014 ip-10-0-135-59 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:49:57.311181 ip-10-0-135-59 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:49:57.311226 ip-10-0-135-59 systemd[1]: kubelet.service: Consumed 2.121s CPU time.
Oct 30 21:49:57.328295 ip-10-0-135-59 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:49:57.407616 ip-10-0-135-59 kubenswrapper[4295]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:57.407616 ip-10-0-135-59 kubenswrapper[4295]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:49:57.407616 ip-10-0-135-59 kubenswrapper[4295]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:57.407616 ip-10-0-135-59 kubenswrapper[4295]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:57.410064 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.409973    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.410064 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410059    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.410064 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410064    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.410064 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410069    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.410064 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410073    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410077    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410082    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410085    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410089    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410093    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410097    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410100    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410104    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410108    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410112    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410116    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410120    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410124    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410128    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410132    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410136    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410139    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410145    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410149    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.410545 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410152    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410156    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410167    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410171    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410175    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410178    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410182    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410186    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410190    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410195    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410199    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410204    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410208    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410213    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410218    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410222    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410227    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410231    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410235    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.411423 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410239    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410244    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410247    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410251    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410255    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410259    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410264    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410268    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410272    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410275    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410279    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410284    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410288    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410295    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410302    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410307    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410311    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410317    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410321    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.412333 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410325    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410330    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410334    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410338    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410342    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410346    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410350    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410354    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410358    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410363    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410367    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410371    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410375    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410379    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410384    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410388    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410392    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410396    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410400    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.413156 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410403    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410407    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410411    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410415    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410419    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410423    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410429    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410433    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410437    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410441    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410446    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410450    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410454    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410458    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410462    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410466    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410473    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410480    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410486    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.414022 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410490    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410494    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410498    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410503    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410509    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.410514    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411850    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411860    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411865    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411869    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411874    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411878    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411882    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411886    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411891    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411895    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411899    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411903    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411907    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411912    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.414878 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411917    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411922    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411926    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411930    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411934    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411938    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411946    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411953    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411958    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411964    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411969    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411975    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411980    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411985    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411990    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411994    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.411998    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412004    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.415804 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412010    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412015    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412019    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412023    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412027    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412031    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412036    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412040    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412044    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412048    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412053    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412057    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412061    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412065    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412070    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412075    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412079    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412083    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412087    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412091    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.416551 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412095    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412099    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412104    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412108    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412112    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412117    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412121    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412132    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412137    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412141    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412145    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412149    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412153    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412158    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412162    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412166    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412170    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412174    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412178    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412182    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.417091 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412186    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412193    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412197    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412202    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412206    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412210    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412214    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412218    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412223    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412227    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412232    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412236    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412240    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412244    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412248    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412252    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412257    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412261    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412265    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.417614 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412269    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412273    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412277    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412281    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412285    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412290    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412294    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412298    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412303    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412308    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412312    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412316    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412322    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412326    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.412330    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413178    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413187    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413192    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413197    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.418111 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413201    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413206    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413211    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413216    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413220    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413224    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413228    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413232    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413237    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413241    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413246    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413251    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413255    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413259    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413264    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413270    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413274    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413279    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413283    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.418622 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413288    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413292    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413300    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413304    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413309    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413313    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413317    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413321    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413326    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413330    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413334    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413338    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413342    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413346    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413351    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413355    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413360    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413365    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413370    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.419112 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413374    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413378    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413382    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413386    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413391    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413395    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413399    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413404    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413408    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413412    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413416    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413420    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413424    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413428    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413432    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413439    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413443    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413447    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413451    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413455    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.429461 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413459    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413464    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413467    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413472    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413476    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413480    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413484    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413488    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413492    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413496    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413501    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413505    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413509    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413513    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413518    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413522    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413526    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413530    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413534    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413538    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.430085 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413542    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413547    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413551    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413555    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413559    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413563    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413567    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413573    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413577    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413581    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413585    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413589    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413594    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413598    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413602    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413607    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413611    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413615    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413621    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413626    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.430557 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413632    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413636    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413642    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.413647    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413803    4295 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413820    4295 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413830    4295 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413840    4295 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413846    4295 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413852    4295 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413858    4295 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413864    4295 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413870    4295 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413875    4295 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413880    4295 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413885    4295 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413890    4295 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413895    4295 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413899    4295 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413905    4295 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413910    4295 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413915    4295 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413922    4295 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:49:57.431062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413927    4295 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413932    4295 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413937    4295 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413942    4295 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413949    4295 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413954    4295 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413959    4295 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413964    4295 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413968    4295 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413973    4295 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413978    4295 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413983    4295 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413987    4295 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.413995    4295 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414000    4295 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414004    4295 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414012    4295 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414017    4295 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414021    4295 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414028    4295 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414033    4295 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414037    4295 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414041    4295 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414047    4295 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:49:57.431777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414054    4295 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414058    4295 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414063    4295 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414069    4295 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414076    4295 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414080    4295 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414085    4295 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414090    4295 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414094    4295 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414099    4295 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414104    4295 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414111    4295 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414115    4295 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414120    4295 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414125    4295 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414130    4295 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414135    4295 flags.go:64] FLAG: --help="false"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414140    4295 flags.go:64] FLAG: --hostname-override="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414145    4295 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414150    4295 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414155    4295 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414160    4295 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414167    4295 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:49:57.432537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414172    4295 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414178    4295 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414182    4295 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414187    4295 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414192    4295 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414197    4295 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414202    4295 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414207    4295 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414211    4295 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414216    4295 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414221    4295 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414225    4295 flags.go:64] FLAG: --lock-file=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414230    4295 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414237    4295 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414243    4295 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414251    4295 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414256    4295 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414261    4295 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414266    4295 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414270    4295 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414276    4295 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414280    4295 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414285    4295 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414291    4295 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414296    4295 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:49:57.433150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414302    4295 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414307    4295 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414311    4295 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414316    4295 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414321    4295 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414326    4295 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414331    4295 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414336    4295 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414349    4295 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414354    4295 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414359    4295 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414364    4295 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414368    4295 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414373    4295 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414378    4295 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414382    4295 flags.go:64] FLAG: --port="10250"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414387    4295 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414391    4295 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01038558fee73058d"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414396    4295 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414401    4295 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414408    4295 flags.go:64] FLAG: --register-node="true"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414413    4295 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414419    4295 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414423    4295 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414427    4295 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414432    4295 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:49:57.433769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414438    4295 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414443    4295 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414447    4295 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414453    4295 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414458    4295 flags.go:64] FLAG: --runonce="false"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414462    4295 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414467    4295 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414472    4295 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414477    4295 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414481    4295 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414486    4295 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414491    4295 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414496    4295 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414501    4295 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414508    4295 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414514    4295 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414518    4295 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414523    4295 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414527    4295 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414532    4295 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414541    4295 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414546    4295 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414550    4295 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414557    4295 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414562    4295 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:49:57.434388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414567    4295 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414573    4295 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414578    4295 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414582    4295 flags.go:64] FLAG: --v="2"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414589    4295 flags.go:64] FLAG: --version="false"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414596    4295 flags.go:64] FLAG: --vmodule=""
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414602    4295 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.414607    4295 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414886    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414894    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414899    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414904    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414908    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414913    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414917    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414921    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414925    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414930    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414935    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414939    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414943    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414951    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.435002 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414956    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414960    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414964    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414968    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414973    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414980    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414986    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414992    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.414999    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415004    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415009    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415016    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415021    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415025    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415030    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415034    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415038    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415043    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.435527 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415047    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415053    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415057    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415061    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415065    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415069    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415073    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415077    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415081    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415086    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415090    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415094    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415099    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415106    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415110    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415115    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415119    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415124    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415128    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415132    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.435972 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415137    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415141    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415145    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415150    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415154    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415160    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415164    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415169    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415173    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415177    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415181    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415186    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415190    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415194    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415199    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415203    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415207    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415211    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415215    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415220    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.436534 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415224    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415228    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415232    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415237    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415241    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415247    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415251    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415256    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415260    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415265    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415269    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415274    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415278    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415282    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415286    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415290    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415295    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415300    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415305    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.437363 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415310    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415314    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415318    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415322    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415326    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415330    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415335    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415339    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415343    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415347    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415352    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415356    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415360    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415364    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415369    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.415409    4295 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:57.438161 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415471    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415479    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415486    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415491    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415495    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415499    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415505    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415509    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415513    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415517    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415521    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415525    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415529    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415533    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415537    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415543    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415548    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415552    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415556    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415560    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.439011 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415564    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415569    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415573    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415577    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415581    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415585    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415590    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415594    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415598    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415602    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415606    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415611    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415616    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415620    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415626    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415630    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415635    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415639    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415643    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415648    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.439689 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415652    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415656    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415681    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415685    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415690    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415694    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415698    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415703    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415707    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415712    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415716    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415720    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415724    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415729    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415733    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415737    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415741    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415746    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415750    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.440193 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415754    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415758    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415762    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415767    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415772    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415776    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415780    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415786    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415790    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415795    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415799    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415804    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415808    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415813    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415818    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415822    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415826    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415830    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415834    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415839    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.440699 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415845    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415849    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415854    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415859    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415863    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415867    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415872    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415876    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415880    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415884    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415888    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415895    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415900    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415906    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415912    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415917    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415922    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415926    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415930    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.441186 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415936    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415940    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415947    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415951    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415955    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415959    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415963    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.415968    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.420055    4295 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.420071    4295 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.420090    4295 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.420094    4295 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420165    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420172    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420175    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420178    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420181    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420184    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420187    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.441655 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420190    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420193    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420196    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420199    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420201    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420204    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420206    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420209    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420212    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420215    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420218    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420221    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420223    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420226    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420228    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420231    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420233    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420236    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420239    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420241    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.442180 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420244    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420246    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420249    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420251    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420254    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420256    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420260    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420262    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420265    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420267    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420269    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420272    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420274    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420276    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420279    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420281    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420283    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420287    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420291    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.442686 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420294    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420297    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420299    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420302    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420305    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420308    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420310    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420313    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420316    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420318    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420320    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420323    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420325    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420329    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420333    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420336    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420338    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420340    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420343    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.443132 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420345    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420348    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420350    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420352    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420355    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420357    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420360    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420362    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420365    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420367    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420369    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420372    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420374    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420377    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420379    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420382    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420385    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420387    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420389    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420392    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.443581 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420394    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420397    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420399    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420401    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420404    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420406    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420408    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420411    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420413    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420416    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420418    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420421    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420423    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420425    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420428    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420430    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420433    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420435    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420438    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.444083 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420440    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420443    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.420469    4295 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.420955    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421381    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421387    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421393    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421398    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421404    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421411    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421422    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421428    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421434    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421439    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421444    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.444543 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421454    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421459    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421463    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421467    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421471    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421474    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421479    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421483    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421488    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421492    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421497    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421502    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421507    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421516    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421521    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421525    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421530    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421534    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421539    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.444918 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421543    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421547    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421552    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421557    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421565    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421569    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421578    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421582    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421587    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421591    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421595    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421599    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421604    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421608    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421612    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421617    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421621    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421626    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421631    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421641    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.445624 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421645    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.445513 ip-10-0-135-59 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421649    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421653    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421680    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421687    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421692    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421697    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421702    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421707    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421715    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421720    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421729    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421734    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421737    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421742    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421746    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421752    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421757    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421761    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421765    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.446762 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421769    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421773    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421777    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421785    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421798    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421840    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421989    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.421996    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422001    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422005    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422009    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422013    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422018    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422023    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422027    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422030    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422034    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422038    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422043    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422046    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.447489 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422050    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422054    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422058    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422062    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422067    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422071    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422075    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422079    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422083    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422087    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422091    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422095    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422100    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422104    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422108    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.448501 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.422158    4295 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422211    4295 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422215    4295 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422219    4295 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422222    4295 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422224    4295 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422227    4295 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422229    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422232    4295 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422235    4295 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422237    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422239    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422242    4295 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422244    4295 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422247    4295 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422249    4295 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422251    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422254    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422256    4295 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422258    4295 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:57.449244 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422261    4295 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422263    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422266    4295 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422268    4295 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422270    4295 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422273    4295 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422275    4295 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422278    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422280    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422283    4295 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422285    4295 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422288    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422291    4295 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422294    4295 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422296    4295 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422299    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422302    4295 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422305    4295 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422308    4295 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:57.450268 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422310    4295 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422313    4295 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422315    4295 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422317    4295 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422320    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422322    4295 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422325    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422329    4295 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422333    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422336    4295 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422339    4295 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422342    4295 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422345    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422347    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422350    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422352    4295 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422355    4295 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422357    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422359    4295 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:57.451075 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422362    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422364    4295 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422367    4295 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422369    4295 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422372    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422375    4295 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422377    4295 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422380    4295 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422383    4295 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422386    4295 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422389    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422391    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422394    4295 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422396    4295 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422399    4295 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422401    4295 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422403    4295 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422406    4295 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422408    4295 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422411    4295 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:57.451889 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422413    4295 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422416    4295 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422419    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422421    4295 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422423    4295 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422426    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422428    4295 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422431    4295 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422434    4295 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422437    4295 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422440    4295 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422444    4295 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422448    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422452    4295 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422468    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422471    4295 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422479    4295 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422483    4295 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422486    4295 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:57.452893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422489    4295 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422492    4295 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422496    4295 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422500    4295 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422502    4295 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422505    4295 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422508    4295 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422510    4295 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422513    4295 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:57.422515    4295 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.422578    4295 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.423559    4295 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.423613    4295 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.423809    4295 server.go:1014] "Starting client certificate rotation"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.423941    4295 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.423984    4295 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:17 +0000 UTC" deadline="2028-05-02 08:20:24.731427222 +0000 UTC"
Oct 30 21:49:57.453703 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.424019    4295 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13186h30m27.307409823s"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.424316    4295 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.425551    4295 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.426932    4295 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.428339    4295 log.go:25] "Validated CRI v1 image API"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.429061    4295 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:49:57.454396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.430514    4295 fs.go:135] Filesystem UUIDs: map[5558aabc-ed6a-4cfa-8117-61ded741325b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ec18faa8-19dc-46dd-914b-f0ac3e23e906:/dev/nvme0n1p3]
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.430528    4295 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0294bcd1-88a6-47de-a415-61451b91c9c2/volumes/kubernetes.io~projected/kube-api-access-xn95r:{mountpoint:/var/lib/kubelet/pods/0294bcd1-88a6-47de-a415-61451b91c9c2/volumes/kubernetes.io~projected/kube-api-access-xn95r major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9/volumes/kubernetes.io~projected/kube-api-access-q6c5r:{mountpoint:/var/lib/kubelet/pods/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9/volumes/kubernetes.io~projected/kube-api-access-q6c5r major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1de1bbac-ecd0-4ac7-8357-71424716cb03/volumes/kubernetes.io~projected/kube-api-access-pvzl4:{mountpoint:/var/lib/kubelet/pods/1de1bbac-ecd0-4ac7-8357-71424716cb03/volumes/kubernetes.io~projected/kube-api-access-pvzl4 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3adc69d5-6bad-412c-abe9-687f28a7a590/volumes/kubernetes.io~projected/kube-api-access-nqs59:{mountpoint:/var/lib/kubelet/pods/3adc69d5-6bad-412c-abe9-687f28a7a590/volumes/kubernetes.io~projected/kube-api-access-nqs59 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bd8c8b1-9d40-4427-9a3c-71a449e6af59/volumes/kubernetes.io~projected/kube-api-access-zbfpt:{mountpoint:/var/lib/kubelet/pods/3bd8c8b1-9d40-4427-9a3c-71a449e6af59/volumes/kubernetes.io~projected/kube-api-access-zbfpt major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72f8a528-1c9b-4f51-8e8f-f298e8982c98/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/72f8a528-1c9b-4f51-8e8f-f298e8982c98/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b478c4f-7211-4b12-ab86-cd866de1ae8a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8b478c4f-7211-4b12-ab86-cd866de1ae8a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~projected/kube-api-access-6qs7w:{mountpoint:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~projected/kube-api-access-6qs7w major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acaa0470-d397-4d38-b674-ff07fee73459/volumes/kubernetes.io~projected/kube-api-access-v2b22:{mountpoint:/var/lib/kubelet/pods/acaa0470-d397-4d38-b674-ff07fee73459/volumes/kubernetes.io~projected/kube-api-access-v2b22 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05/volumes/kubernetes.io~projected/kube-api-access-lfbl6:{mountpoint:/var/lib/kubelet/pods/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05/volumes/kubernetes.io~projected/kube-api-access-lfbl6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df135a70-eac2-457e-b470-bd45b99875c6/volumes/kubernetes.io~projected/kube-api-access-k7skz:{mountpoint:/var/lib/kubelet/pods/df135a70-eac2-457e-b470-bd45b99875c6/volumes/kubernetes.io~projected/kube-api-access-k7skz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~projected/kube-api-access-2z7z5:{mountpoint:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~projected/kube-api-access-2z7z5 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/eebb0c775093daeb40f11840cf05416bdd31c454e405f4084f068cc377aca94b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/21e4a65309b2e384f53b358e15164c6b3fea4425092c556e61d0a509f8113255/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fe5aca5f4d96340ead0eab9d36e1e193a99346658937211f6f30770ec2ac256c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/d4c43a642bb029ad242389cf1058ddb3fd4e8104ae9bbc560cecaebbf4e8e894/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b8426bd9f77fd8b76d04b46fe99fcd3c4c36e85381cddb9f872a199d749c9bd1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4881c318e7e99e6a1158240b94415f8d1d789e328ae58da37e11fde8213fcfe6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6149b28ea18533809ecb668cd55239ed99fe0fa4132619fbf10b719fa294b2f7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f3c0d3f29a2cc82a4252d8940be5116ac76778def32edf06c35e0a271c9ff5fb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c62dfe0cf011c1ed4506547cb8d06d9fb9b6fc3fcdc8839d2a6e5232b5804847/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/22de5298b65def113eecdd026bb06a889933bf40ab2ed34d7dfcb799b3e6dd75/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/7c941f1f734ec16a6c0014526163e6a37832284af6a7bc9bd6af7d66da22e0f4/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/edbfa786fa4dbe8508ba80fdf8cc3e4ff95f77c4d9e587897dca0dc3f874d4fb/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2c722f10bdb72fbf2bf9c7eea90e1097e6c6bff38ec0b8f6ca049969252989bc/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/152c62a955737e2619f970da0002b0ebb1ce75c903e55ddefb13a0f125014ee0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/faa8c6c99a861565a0607ec7b34a192e932829c1837b84f170fefec7e33bdb61/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9105bbf25e64e40e2aa84cd520587d72e6d5b61df8756a51bf252fe83ebc06df/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b71bfc5fc9bc0b7f050e2c18af1594e2d50ea49a9cc3303784a70b75ccd05d24/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bb6a69e9fcdc3db73d77c866293225072b9f9a391bb8ccc360287ee8e68afedb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/36b33edcd19e6c961fac359d7d82f859984241651f6501ea6ab138f70ff5a00a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d47a3fdfb1cd9b024f87a2487d5d07e9ef77b264fd830d55d897203a01df5241/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/21500f90a8a0503a9bd2b12c07d363cbfe08b3c0e3bc3e96f65d0e6f82fa0705/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/af1d0db84a1c7c6fc2f982dfef5772478598e33acb815ec6e30e6817a9c7eb88/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/fbc22c7b48c4568bca3f2fd3c8c7e8b80b7153b07c2f00acbc6c0d8ee062edff/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/0a09a41b6c27a1231db582aea2e002742aca67c428cfcbd48a5e88200bf51769/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7bfabae7717aa63cf28596b9f6836e61eb6777305342dba5a1d743b042a2aecf/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/9a2151686790497c092a6ecbd5bfea1cf928bf4cf8cbf435aa701882cbcb004d/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/67301dd841728ad9d732e5abe5969a30dab78341ab660401b126f76c59eb90c0/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/1f77c05ab426ce62e9ac1d4951c65cc52299874cd90bf19bd011e1ea2372729b/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/043d53d840c8bdccb8b9f904d2b9795175a7c31fc2dda8223fcd14ae03704285/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/4404ef4b67601852538b0da4c0d92515fbcbded41849bfc7f34e959e2a925e76/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/bec754a280ca6bef0bb6ae07e2168df1eea7f55020326ac1d183e34f7b7d6c11/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/cd98dfefa394eb459629217bab7a93d8977728734525c20a9337a4f97961c5fe/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/df14330d354361050cf70a7ccadb73c18f6335af84de5783fa5846eec1aa61a8/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/704951ef315b337db729a92fe31f4c7a7344724b5e52753538b38124f0173ba9/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cf1ed9cecd316a8980e2c579fd136c727377d9dd1653016505d13a7696df302c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436043    4295 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:57.435586678 +0000 UTC m=+0.078180457 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100303 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec3f90aec1864ba74af74050015f5 SystemUUID:ec2ec3f9-0aec-1864-ba74-af74050015f5 BootID:572fbd6a-ffdd-4c21-b61c-0d3bdf6a4938 Filesystems:[{Device:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf98a6a751c54de1380755fb3121535f5a255eaf3617800a87ccf3bf9ea1f448/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14c5d0e1ea602a26169fc154fb33f24020c2af67d7b7f3fdabcc7bdc4d165258/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6956720d098428b941d45c27f50272fb0c7bcf5f021b1a8ddb9486536a2f19dd/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9/volumes/kubernetes.io~projected/kube-api-access-q6c5r DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1de1bbac-ecd0-4ac7-8357-71424716cb03/volumes/kubernetes.io~projected/kube-api-access-pvzl4 DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/108c6f297ca9183b4c2365ac4922e3204a8df97c5ad3be2f8c28796f5afb0a3e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0fc3590d549cfd16dc5ac45d93633b348b611b767fc6015f39bfaa10b1d2f2d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~projected/kube-api-access-2z7z5 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0f5e49-2f17-4384-b1f4-32521160dd20/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/72f8a528-1c9b-4f51-8e8f-f298e8982c98/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/acaa0470-d397-4d38-b674-ff07fee73459/volumes/kubernetes.io~projected/kube-api-access-v2b22 DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d37bdc1-5888-4d01-bd66-90f17ef65d8e/volumes/kubernetes.io~projected/kube-api-access-6qs7w DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/293cc49b6c440569520569d39c1f2b4cdf4518ff0f3cb4a0ad3ababad74285ea/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df135a70-eac2-457e-b470-bd45b99875c6/volumes/kubernetes.io~projected/kube-api-access-k7skz DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/540a891dde9335ebf7ccee5bde04767ec2dfbf98b5f7e3d22c66ea64cddf3934/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e047238625742361cc12c5f1f323a278292dec7feb14ca708703229618ce1c5/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05/volumes/kubernetes.io~projected/kube-api-access-lfbl6 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bd8c8b1-9d40-4427-9a3c-71a449e6af59/volumes/kubernetes.io~projected/kube-api-access-zbfpt DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0294bcd1-88a6-47de-a415-61451b91c9c2/volumes/kubernetes.io~projected/kube-api-access-xn95r DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b478c4f-7211-4b12-ab86-cd866de1ae8a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a18d69f4419de25ddcfbf81b49162ea3a46e8d9f31cf8e1b3dbd8dbca80edc13/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3adc69d5-6bad-412c-abe9-687f28a7a590/volumes/kubernetes.io~projected/kube-api-access-nqs59 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ccde1491aaa61f7bdeb2557250b4d86a627a2231dc96c33a320f4ca169ad22/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:a18d69f4419de25 MacAddress:86:68:88:ee:3f:a7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:51:86:ec:78:97 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:68:60:92:b1:b5 Speed:0 Mtu:8901} {Name:e0fc3590d549cfd MacAddress:2a:59:44:cc:52:3d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:51:86:ec:78:97 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:38:4b:09:3e:fe Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:09:83:23:ff:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436267    4295 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436307    4295 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436640    4295 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436686    4295 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-59.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436867    4295 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436879    4295 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436895    4295 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.436919    4295 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437031    4295 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437227    4295 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437359    4295 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437407    4295 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:49:57.454677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437425    4295 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437438    4295 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437452    4295 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.437464    4295 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.438411    4295 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.438428    4295 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.439633    4295 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441319    4295 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441328    4295 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441423    4295 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441592    4295 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441623    4295 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441729    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441741    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441748    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441754    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441760    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441765    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441771    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441776    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441783    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441788    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441797    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441806    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441835    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.441841    4295 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.444998    4295 server.go:1263] "Started kubelet"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.445092    4295 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.445100    4295 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.445171    4295 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.445494    4295 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.446329    4295 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447566    4295 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447585    4295 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447757    4295 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:18 +0000 UTC" deadline="2028-04-25 15:32:45.485526586 +0000 UTC"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447790    4295 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13025h42m48.037739841s"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447820    4295 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447826    4295 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.447901    4295 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449431    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" volumeName="kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449458    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f8a528-1c9b-4f51-8e8f-f298e8982c98" volumeName="kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449470    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d37bdc1-5888-4d01-bd66-90f17ef65d8e" volumeName="kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449483    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acaa0470-d397-4d38-b674-ff07fee73459" volumeName="kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449494    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0f5e49-2f17-4384-b1f4-32521160dd20" volumeName="kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449516    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0f5e49-2f17-4384-b1f4-32521160dd20" volumeName="kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449528    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9" volumeName="kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449541    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f8a528-1c9b-4f51-8e8f-f298e8982c98" volumeName="kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449557    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d37bdc1-5888-4d01-bd66-90f17ef65d8e" volumeName="kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449570    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94e83dd-993e-4e5c-ab7f-ef4db6f57f05" volumeName="kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449596    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0f5e49-2f17-4384-b1f4-32521160dd20" volumeName="kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449611    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0294bcd1-88a6-47de-a415-61451b91c9c2" volumeName="kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449622    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1de1bbac-ecd0-4ac7-8357-71424716cb03" volumeName="kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449632    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" volumeName="kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449644    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" volumeName="kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449683    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acaa0470-d397-4d38-b674-ff07fee73459" volumeName="kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449699    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94e83dd-993e-4e5c-ab7f-ef4db6f57f05" volumeName="kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449711    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0f5e49-2f17-4384-b1f4-32521160dd20" volumeName="kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449723    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0f5e49-2f17-4384-b1f4-32521160dd20" volumeName="kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449737    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0294bcd1-88a6-47de-a415-61451b91c9c2" volumeName="kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449749    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0294bcd1-88a6-47de-a415-61451b91c9c2" volumeName="kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449762    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" volumeName="kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449773    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b478c4f-7211-4b12-ab86-cd866de1ae8a" volumeName="kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449790    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d37bdc1-5888-4d01-bd66-90f17ef65d8e" volumeName="kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449800    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df135a70-eac2-457e-b470-bd45b99875c6" volumeName="kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449810    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9" volumeName="kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449822    4295 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3adc69d5-6bad-412c-abe9-687f28a7a590" volumeName="kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59" seLinuxMountContext=""
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449835    4295 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.449843    4295 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.450286    4295 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found"
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.451699    4295 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.451714    4295 factory.go:55] Registering systemd factory
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.451723    4295 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.451989    4295 factory.go:153] Registering CRI-O factory
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.452002    4295 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.452019    4295 factory.go:103] Registering Raw factory
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.452034    4295 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:49:57.455619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.453015    4295 manager.go:319] Starting recovery of all containers
Oct 30 21:49:57.483984 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.483951    4295 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:49:57.485516 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.485496    4295 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:49:57.485516 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.485518    4295 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:49:57.485634 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.485536    4295 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:49:57.485634 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.485574    4295 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:49:57.502585 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.502547    4295 generic.go:356] "Generic (PLEG): container finished" podID="1c256e02c33d90178c37b95ddf27e766" containerID="e785cf876a03872ab6ea0d59b570af456330f03b5eb624f4356782885bb75503" exitCode=0
Oct 30 21:49:57.511565 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511543    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="e398f4ee763eac94725169452822068274ca27a12789b23752b2270691f38afc" exitCode=0
Oct 30 21:49:57.511565 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511564    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="7af27c842272ef86b11dd8d27b360947590a8f3627f8df7853481ae3c0c6acd9" exitCode=0
Oct 30 21:49:57.511746 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511572    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="16ecf9f64d3f729318a6fc4570a1bad98a318a2b5bd8e9cc9c65d7158e5ada3f" exitCode=0
Oct 30 21:49:57.511746 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511580    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="11bef436afe22c212c32b04498c395c1ff2044b4579d232a519d40f8539d8294" exitCode=0
Oct 30 21:49:57.511746 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511587    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="987527a9780397cf08383ca751a35ad54008456334ffb1023817421127fead9c" exitCode=0
Oct 30 21:49:57.511746 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.511596    4295 generic.go:356] "Generic (PLEG): container finished" podID="3bd8c8b1-9d40-4427-9a3c-71a449e6af59" containerID="a947e9e465c9d61d1b89a867565c0fdbe53a32598e2763cd9a8452e5bb8f8dd8" exitCode=0
Oct 30 21:49:57.512389 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.512376    4295 manager.go:324] Recovery completed
Oct 30 21:49:57.526082 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.526067    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.527063 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.527041    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.527136 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.527072    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.527136 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.527083    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.528206 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528196    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.528268 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528199    4295 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:49:57.528313 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528274    4295 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:49:57.528313 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528291    4295 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:49:57.528409 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528396    4295 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:49:57.528438 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528410    4295 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:49:57.528438 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528426    4295 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:49:57.528438 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528431    4295 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:49:57.528438 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528436    4295 policy_none.go:50] "Start"
Oct 30 21:49:57.528540 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528447    4295 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:49:57.528540 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528456    4295 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:49:57.528594 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528572    4295 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:49:57.528594 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528579    4295 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:49:57.528594 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528584    4295 policy_none.go:44] "Start"
Oct 30 21:49:57.528924 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528911    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.528987 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528937    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.528987 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.528950    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.532364 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.532342    4295 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:49:57.535505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535490    4295 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:49:57.535575 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.535511    4295 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:49:57.535575 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535526    4295 server.go:87] "Starting device plugin registration server"
Oct 30 21:49:57.535803 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535791    4295 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:49:57.535879 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535802    4295 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:49:57.535930 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535899    4295 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:49:57.536005 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.535972    4295 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:49:57.536005 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.536005    4295 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:49:57.536005 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.536011    4295 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:49:57.536188 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.536058    4295 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:57.535997439Z","UUID":"d95b7d06-73d2-43fa-bdf0-08add4474540","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:49:57.537196 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.537175    4295 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:49:57.537288 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.537210    4295 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-59.us-west-1.compute.internal\" not found"
Oct 30 21:49:57.537471 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.537454    4295 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:49:57.537561 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.537477    4295 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:49:57.586623 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.586141    4295 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"]
Oct 30 21:49:57.586747 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.586684    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.587521 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.587506    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.587590 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.587532    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.587590 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.587544    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.587803 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.587792    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.587948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.587936    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.589942 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.589924    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.590038 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.589957    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.590038 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.589968    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.590038 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.589929    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.590038 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590031    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.590171 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590046    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.590171 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.590118    4295 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.590278 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590252    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerStarted","Data":"d8943ba3d1a08a8793905bbc31b4fa3dcfa36197cb4e3b7cc48dada30f7e23d4"}
Oct 30 21:49:57.590315 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590292    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerDied","Data":"e785cf876a03872ab6ea0d59b570af456330f03b5eb624f4356782885bb75503"}
Oct 30 21:49:57.590343 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590314    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"1c256e02c33d90178c37b95ddf27e766","Type":"ContainerStarted","Data":"3b3727ec988e6fb66a68d559b7eff8314186703f3779b7bcfaeddae030dde18f"}
Oct 30 21:49:57.590373 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590359    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.590401 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590372    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"106668d41ee4901fe4ed1e1a301dc3c3","Type":"ContainerStarted","Data":"9533cf65f618d712a42b04087ce7535c5decdbf02ea90fe3fde052339c5f2632"}
Oct 30 21:49:57.590401 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.590392    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal" event={"ID":"106668d41ee4901fe4ed1e1a301dc3c3","Type":"ContainerStarted","Data":"7a154b8cabb746eb3f0ca463081e9052e19e241c52beb5bc43d620909c42ec6e"}
Oct 30 21:49:57.591110 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.591097    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.591185 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.591118    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.591185 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.591128    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.591268 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.591234    4295 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-59.us-west-1.compute.internal\" not found" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.636401 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.636374    4295 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:57.637762 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.637742    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:57.637856 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.637774    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:57.637856 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.637785    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:57.637856 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.637803    4295 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.654315 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.654291    4295 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:49:57.701641 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.701615    4295 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:57.704679 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.704642    4295 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:57.704782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.704736    4295 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:57.751007 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.750984    4295 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.752251 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.752233    4295 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:57.755046 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.755027    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.755116 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.755055    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.755116 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.755074    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.805875 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.805854    4295 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.805937 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.805922    4295 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.806442 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.806420    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 21:49:57.806546 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.806456    4295 setters.go:546] "Node became not ready" node="ip-10-0-135-59.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:49:57Z","lastTransitionTime":"2026-10-30T21:49:57Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 21:49:57.807003 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.806985    4295 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-59.us-west-1.compute.internal" is owned by different node (oldNodeID="dbb2e8f8-6349-47e8-b646-b18d8341dbc5", newNodeID=""), cleaning up...
Oct 30 21:49:57.813791 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.813775    4295 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:57.813863 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.813824    4295 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.813863 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.813841    4295 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856134 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856076    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856134 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856102    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856134 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856117    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856283 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856163    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856283 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856187    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c256e02c33d90178c37b95ddf27e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"1c256e02c33d90178c37b95ddf27e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.856283 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.856199    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/106668d41ee4901fe4ed1e1a301dc3c3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" (UID: \"106668d41ee4901fe4ed1e1a301dc3c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:57.878083 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:57.878060    4295 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:57.878187 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:57.878110    4295 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:58.438242 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.438212    4295 apiserver.go:52] "Watching apiserver"
Oct 30 21:49:58.520506 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.520478    4295 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:58.520698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.520558    4295 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:58.545285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.545265    4295 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:49:58.545689 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.545652    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-2vmrb","kube-system/global-pull-secret-syncer-jhchm","kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-hxvm4","openshift-multus/network-metrics-daemon-j9lzw","openshift-network-operator/iptables-alerter-md9xp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d","openshift-cluster-node-tuning-operator/tuned-ksctb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-6qwzk","kube-system/konnectivity-agent-nj82c","openshift-image-registry/node-ca-zpqg6","openshift-ovn-kubernetes/ovnkube-node-wq75f","openshift-dns/node-resolver-t8f8p"]
Oct 30 21:49:58.546748 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.546727    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.560037 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560013    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.560151 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560053    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs7w\" (UniqueName: \"kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560151 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560077    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b22\" (UniqueName: \"kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.560151 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560124    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.560319 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560194    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn95r\" (UniqueName: \"kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.560319 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560283    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbl6\" (UniqueName: \"kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.560388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560311    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560388 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560348    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560486 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560446    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.560541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560494    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560522    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560617 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560548    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.560646 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560616    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.560701 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560687    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-tmp\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.560751 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560693    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzl4\" (UniqueName: \"kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.560803 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560734    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:58.560803 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.560773    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.561135 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561109    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.561247 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561160    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.561247 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561193    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.561360 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561269    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.561360 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561306    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-tuned\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.561360 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561346    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:58.561483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561396    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.561483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561431    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.561483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561466    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.561634 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561499    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.561634 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561531    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.561634 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561559    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.561800 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561692    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.561800 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561734    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.561800 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561767    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.561800 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561798    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.561959 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561831    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:58.561959 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561862    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.561959 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561893    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.561959 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561925    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561956    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.561983    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562012    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562041    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562074    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfpt\" (UniqueName: \"kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.562109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562098    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562125    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562157    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562187    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562214    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562239    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562269    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs59\" (UniqueName: \"kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562297    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.562333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562325    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562356    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562391    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562414    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562444    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562473    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562502    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562529    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562555    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.562647 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.562584    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.585879 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.585858    4295 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:58.585977 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:58.585906    4295 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:58.586702 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.586689    4295 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:58.586734 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:49:58.586720    4295 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-59.us-west-1.compute.internal"
Oct 30 21:49:58.603159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.603145    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:49:58.603159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.603153    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:49:58.603235 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.603188    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:49:58.603268 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.603259    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.603299 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.603273    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:49:58.604563 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.604540    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.604640 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.604548    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-iptables-alerter-script\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.608425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608411    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:49:58.608483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608465    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.608535 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608514    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:49:58.608535 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608528    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.608610 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608530    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.608610 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.608561    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.610890 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.610875    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-cni-binary-copy\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.610926 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.610887    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-daemon-config\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.611964 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.611948    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.613029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.613016    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.616930 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616910    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616918    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:49:58.617017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616933    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:49:58.617017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616939    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616938    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:49:58.617017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.616971    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.617193 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617020    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\""
Oct 30 21:49:58.617193 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617049    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\""
Oct 30 21:49:58.617193 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617125    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617630 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617611    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.617630 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617614    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.617777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617633    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:49:58.617777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617628    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617651    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617748    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.617777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617769    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617876    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617878    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617920    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617930    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617946    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.617969 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617964    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617980    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617988    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618002    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618004    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618012    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.617991    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618031    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618067    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618087    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\""
Oct 30 21:49:58.618146 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.618147    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:49:58.622441 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.622424    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a528-1c9b-4f51-8e8f-f298e8982c98-konnectivity-ca\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:58.622521 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.622442    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b478c4f-7211-4b12-ab86-cd866de1ae8a-original-pull-secret\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.623514 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.623500    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72f8a528-1c9b-4f51-8e8f-f298e8982c98-agent-certs\") pod \"konnectivity-agent-nj82c\" (UID: \"72f8a528-1c9b-4f51-8e8f-f298e8982c98\") " pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:49:58.623846 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.623828    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acaa0470-d397-4d38-b674-ff07fee73459-serviceca\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.639071 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.639041    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs7w\" (UniqueName: \"kubernetes.io/projected/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-kube-api-access-6qs7w\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.639139 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.639045    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbl6\" (UniqueName: \"kubernetes.io/projected/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-kube-api-access-lfbl6\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.639174 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.639136    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn95r\" (UniqueName: \"kubernetes.io/projected/0294bcd1-88a6-47de-a415-61451b91c9c2-kube-api-access-xn95r\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.646776 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.646760    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzl4\" (UniqueName: \"kubernetes.io/projected/1de1bbac-ecd0-4ac7-8357-71424716cb03-kube-api-access-pvzl4\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.648940 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.648922    4295 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:49:58.649264 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.649245    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfpt\" (UniqueName: \"kubernetes.io/projected/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-kube-api-access-zbfpt\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.649427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.649408    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs59\" (UniqueName: \"kubernetes.io/projected/3adc69d5-6bad-412c-abe9-687f28a7a590-kube-api-access-nqs59\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.649633 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.649617    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7skz\" (UniqueName: \"kubernetes.io/projected/df135a70-eac2-457e-b470-bd45b99875c6-kube-api-access-k7skz\") pod \"network-check-target-6qwzk\" (UID: \"df135a70-eac2-457e-b470-bd45b99875c6\") " pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:49:58.650265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.650249    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b22\" (UniqueName: \"kubernetes.io/projected/acaa0470-d397-4d38-b674-ff07fee73459-kube-api-access-v2b22\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.663198 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663178    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663296 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663209    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663296 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663235    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663296 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663260    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663299    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-sys\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663290    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-systemd\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663325    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663349    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663368    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663388    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-host\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663390    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.663425 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663423    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-kubelet-config\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663438    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663442    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-k8s-cni-cncf-io\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663466    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663489    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663499    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-socket-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663523    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-socket-dir-parent\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663530    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663543    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovn-node-metrics-cert\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663553    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b478c4f-7211-4b12-ab86-cd866de1ae8a-dbus\") pod \"global-pull-secret-syncer-jhchm\" (UID: \"8b478c4f-7211-4b12-ab86-cd866de1ae8a\") " pod="kube-system/global-pull-secret-syncer-jhchm"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663559    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663587    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c5r\" (UniqueName: \"kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663688    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663705    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663721    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663737    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663760    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663774    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c94e83dd-993e-4e5c-ab7f-ef4db6f57f05-host-slash\") pod \"iptables-alerter-md9xp\" (UID: \"c94e83dd-993e-4e5c-ab7f-ef4db6f57f05\") " pod="openshift-network-operator/iptables-alerter-md9xp"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663785    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663800    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-system-cni-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663806    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-hostroot\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663817    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-multus-certs\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663810    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663839    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-registration-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663853    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663859    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-script-lib\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663894    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663909    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-conf\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663948    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-modprobe-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663954    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.663978    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysconfig\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664007    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664091    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664112    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664123    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-bin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664141    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664158    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664143    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-ovnkube-config\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664170    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664205    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-cnibin\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664234    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664273    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664280    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-kubernetes\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664306    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664314    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea0f5e49-2f17-4384-b1f4-32521160dd20-env-overrides\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664335    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664362    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7z5\" (UniqueName: \"kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664378    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664424    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.664495 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664467    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-tmp-dir\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664519    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664527    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-system-cni-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664548    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664574    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-cnibin\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664584    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664600    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-sys-fs\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664612    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664693    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664718    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-var-lib-kubelet\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664720    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-lib-modules\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664752    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664775    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664790    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-run-netns\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664791    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664808    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acaa0470-d397-4d38-b674-ff07fee73459-host\") pod \"node-ca-zpqg6\" (UID: \"acaa0470-d397-4d38-b674-ff07fee73459\") " pod="openshift-image-registry/node-ca-zpqg6"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664821    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664846    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664870    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664926    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-multus-conf-dir\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664931    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664934    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664966    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664979    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-run\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664975    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d37bdc1-5888-4d01-bd66-90f17ef65d8e-etc-sysctl-d\") pod \"tuned-ksctb\" (UID: \"9d37bdc1-5888-4d01-bd66-90f17ef65d8e\") " pod="openshift-cluster-node-tuning-operator/tuned-ksctb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.664992    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665038    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665064    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665088    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665098    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-etc-kubernetes\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665115    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665142    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665152    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd8c8b1-9d40-4427-9a3c-71a449e6af59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxvm4\" (UID: \"3bd8c8b1-9d40-4427-9a3c-71a449e6af59\") " pod="openshift-multus/multus-additional-cni-plugins-hxvm4"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665165    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665197    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665213    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665229    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665218    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-cni-multus\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665266    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665291    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665291    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3adc69d5-6bad-412c-abe9-687f28a7a590-device-dir\") pod \"aws-ebs-csi-driver-node-s8v9d\" (UID: \"3adc69d5-6bad-412c-abe9-687f28a7a590\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8v9d"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665294    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665347    4295 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:49:58.665564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.665360    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0294bcd1-88a6-47de-a415-61451b91c9c2-host-var-lib-kubelet\") pod \"multus-2vmrb\" (UID: \"0294bcd1-88a6-47de-a415-61451b91c9c2\") " pod="openshift-multus/multus-2vmrb"
Oct 30 21:49:58.667403 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.667377    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de1bbac-ecd0-4ac7-8357-71424716cb03-metrics-certs\") pod \"network-metrics-daemon-j9lzw\" (UID: \"1de1bbac-ecd0-4ac7-8357-71424716cb03\") " pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:58.726168 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.726087    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c5r\" (UniqueName: \"kubernetes.io/projected/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-kube-api-access-q6c5r\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.726780 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.726764    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7z5\" (UniqueName: \"kubernetes.io/projected/ea0f5e49-2f17-4384-b1f4-32521160dd20-kube-api-access-2z7z5\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766182 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766157    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766185    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766203    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766217    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766231    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766249    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766261    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-netd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766267    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766290    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-node-log\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766301    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-netns\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766311    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766312    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-kubelet\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766330    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766293    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-systemd\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766349    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-ovn\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766352    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-slash\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766355    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766406    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766422    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766446    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766374    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-run-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766473    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766497    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766501    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-cni-bin\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.766500 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766471    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766513    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766515    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-var-lib-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766535    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766532    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-systemd-units\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766494    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9-hosts-file\") pod \"node-resolver-t8f8p\" (UID: \"0c1bf1e8-1b8d-4781-ae6f-ac2010aad2d9\") " pod="openshift-dns/node-resolver-t8f8p"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766597    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-log-socket\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.767079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.766640    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea0f5e49-2f17-4384-b1f4-32521160dd20-etc-openvswitch\") pod \"ovnkube-node-wq75f\" (UID: \"ea0f5e49-2f17-4384-b1f4-32521160dd20\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:49:58.847293 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:58.847273    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9lzw"
Oct 30 21:49:59.248637 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:59.248604    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j9lzw"]
Oct 30 21:49:59.252204 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:49:59.252170    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de1bbac_ecd0_4ac7_8357_71424716cb03.slice/crio-7c7e52a242e328c334711ecf6a253eed06351ca6873db3fd2450362486f1c8e6 WatchSource:0}: Error finding container 7c7e52a242e328c334711ecf6a253eed06351ca6873db3fd2450362486f1c8e6: Status 404 returned error can't find the container with id 7c7e52a242e328c334711ecf6a253eed06351ca6873db3fd2450362486f1c8e6
Oct 30 21:49:59.254253 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:59.254238    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:59.523327 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:49:59.523239    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9lzw" event={"ID":"1de1bbac-ecd0-4ac7-8357-71424716cb03","Type":"ContainerStarted","Data":"7c7e52a242e328c334711ecf6a253eed06351ca6873db3fd2450362486f1c8e6"}
Oct 30 21:50:01.528989 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:01.528957    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9lzw" event={"ID":"1de1bbac-ecd0-4ac7-8357-71424716cb03","Type":"ContainerStarted","Data":"0620ab70bc221578572ee4c9edb6ce24c20ce59924ddabc5575230bfc0bd4f0d"}
Oct 30 21:50:01.529395 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:01.529002    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9lzw" event={"ID":"1de1bbac-ecd0-4ac7-8357-71424716cb03","Type":"ContainerStarted","Data":"a7c38577689900e49a964dc74076d28d0f74a037d4407030b92a6f4bee11a29f"}
Oct 30 21:50:02.478395 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:02.478367    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:50:03.324681 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:03.324633    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:50:04.666446 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:04.666419    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:50:04.666925 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:04.666683    4295 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:50:07.303696 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:07.303653    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq75f"
Oct 30 21:50:08.008892 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:08.008861    4295 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-59.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:50:09.931131 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.931103    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"]
Oct 30 21:50:09.932818 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.932803    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:09.945099 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.945082    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-s7g72"]
Oct 30 21:50:09.946399 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.946386    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:09.968397 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.968377    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:09.969726 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.969713    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:09.996338 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.996319    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9fl7p\""
Oct 30 21:50:09.996454 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:09.996399    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:50:10.011801 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.011776    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:50:10.012851 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.012829    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5hv"]
Oct 30 21:50:10.014630 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.014611    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.023855 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.023836    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vj5bd"]
Oct 30 21:50:10.025576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.025561    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.030356 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030338    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48bv\" (UniqueName: \"kubernetes.io/projected/80d01ed3-19c8-43e9-802d-da4f6a080e69-kube-api-access-p48bv\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.030435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030368    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030396    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030414    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030432    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjgs\" (UniqueName: \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030571 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030481    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030571 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030503    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80d01ed3-19c8-43e9-802d-da4f6a080e69-tmp\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.030571 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030523    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030571 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030549    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.030789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.030690    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e767adc-6d0b-4e96-8b5b-9c5384fc5a07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7nsmt\" (UID: \"8e767adc-6d0b-4e96-8b5b-9c5384fc5a07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:10.048874 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.048842    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:50:10.048966 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.048883    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:50:10.048966 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.048932    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:50:10.049560 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.049544    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:50:10.067642 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.067618    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:10.068182 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.068161    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s7g72"]
Oct 30 21:50:10.075850 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.075832    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:50:10.075937 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.075837    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:50:10.076419 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.076405    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:50:10.092296 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.092277    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vj5bd"]
Oct 30 21:50:10.093256 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.093239    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:50:10.099387 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099370    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:50:10.099541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099468    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c6mrn\""
Oct 30 21:50:10.099541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099502    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9l2vj\""
Oct 30 21:50:10.099541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099526    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:50:10.099541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099526    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:50:10.099730 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099547    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-27xvg\""
Oct 30 21:50:10.099730 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.099578    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lwnsl\""
Oct 30 21:50:10.102833 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.102819    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:50:10.114733 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.114711    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vzkxs"]
Oct 30 21:50:10.116408 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.116393    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.117682 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.117646    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cd75cdd8-lfzhm"]
Oct 30 21:50:10.119126 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.119110    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.131078 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131059    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80d01ed3-19c8-43e9-802d-da4f6a080e69-tmp\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.131156 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131089    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.131156 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131105    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdvf\" (UniqueName: \"kubernetes.io/projected/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-api-access-jwdvf\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.131236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131170    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131187    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-config-volume\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.131236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131201    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9w68\" (UniqueName: \"kubernetes.io/projected/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-kube-api-access-w9w68\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.131236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131218    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.131236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131231    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-metrics-tls\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131247    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131326    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80d01ed3-19c8-43e9-802d-da4f6a080e69-tmp\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131343    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e767adc-6d0b-4e96-8b5b-9c5384fc5a07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7nsmt\" (UID: \"8e767adc-6d0b-4e96-8b5b-9c5384fc5a07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131383    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48bv\" (UniqueName: \"kubernetes.io/projected/80d01ed3-19c8-43e9-802d-da4f6a080e69-kube-api-access-p48bv\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131411    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131435    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131462    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-crio-socket\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131493    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131517    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-data-volume\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131545    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjgs\" (UniqueName: \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131567    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-tmp-dir\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131598    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.131823 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131633    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.132108 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.131995    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.132155 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.132096    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.132189 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.132152    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.133902 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.133880    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.134124 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.134103    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.134163 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.134138    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e767adc-6d0b-4e96-8b5b-9c5384fc5a07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7nsmt\" (UID: \"8e767adc-6d0b-4e96-8b5b-9c5384fc5a07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:10.135560 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.135540    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5hv"]
Oct 30 21:50:10.142332 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.142311    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"]
Oct 30 21:50:10.173929 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.173910    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:50:10.174256 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.174243    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:50:10.179932 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.179914    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:50:10.192026 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.191972    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzkxs"]
Oct 30 21:50:10.208725 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.208707    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd75cdd8-lfzhm"]
Oct 30 21:50:10.214685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.214644    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:50:10.214772 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.214725    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vjtf6\""
Oct 30 21:50:10.224264 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.224242    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:50:10.224353 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.224341    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:50:10.224783 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.224769    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:50:10.224783 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.224776    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-84vh4\""
Oct 30 21:50:10.231409 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.231392    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48bv\" (UniqueName: \"kubernetes.io/projected/80d01ed3-19c8-43e9-802d-da4f6a080e69-kube-api-access-p48bv\") pod \"downloads-6dd789c8b7-s7g72\" (UID: \"80d01ed3-19c8-43e9-802d-da4f6a080e69\") " pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.231521 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.231504    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjgs\" (UniqueName: \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\") pod \"console-68f958b5dd-rrgpw\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") " pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.232236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232207    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-image-registry-private-configuration\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.232342 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232240    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.232342 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232258    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-data-volume\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.232342 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232274    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9w68\" (UniqueName: \"kubernetes.io/projected/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-kube-api-access-w9w68\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.232523 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232502    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-registry-tls\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.232581 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232535    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-data-volume\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.232581 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232561    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0882592c-76ee-4747-8c81-e62c4fe88a76-cert\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.232704 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232591    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-trusted-ca\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.232704 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232639    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-installation-pull-secrets\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.232820 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232709    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9j8\" (UniqueName: \"kubernetes.io/projected/0882592c-76ee-4747-8c81-e62c4fe88a76-kube-api-access-8n9j8\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.232820 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232730    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-registry-certificates\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.232820 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232767    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-crio-socket\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.232820 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232802    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-tmp-dir\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232832    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdvf\" (UniqueName: \"kubernetes.io/projected/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-api-access-jwdvf\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232854    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpwq\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-kube-api-access-wbpwq\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232861    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-crio-socket\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232831    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232878    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232933    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23526c30-5d35-42a2-a14e-3525b1dde115-ca-trust-extracted\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.233008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.232964    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-config-volume\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.233255 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.233064    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-tmp-dir\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.233255 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.233070    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-bound-sa-token\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.233255 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.233127    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-metrics-tls\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.233463 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.233449    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-config-volume\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.235038 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.235019    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.235499 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.235480    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-metrics-tls\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.242236 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.242220    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:10.256334 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.256318    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:10.279907 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.278747    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:10.316910 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.316867    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9w68\" (UniqueName: \"kubernetes.io/projected/c38de308-9d4a-46e7-80f9-59e5a61a2fdc-kube-api-access-w9w68\") pod \"dns-default-vj5bd\" (UID: \"c38de308-9d4a-46e7-80f9-59e5a61a2fdc\") " pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.317037 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.316988    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdvf\" (UniqueName: \"kubernetes.io/projected/5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6-kube-api-access-jwdvf\") pod \"insights-runtime-extractor-nm5hv\" (UID: \"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6\") " pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.326861 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.326837    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nm5hv"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334598    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-image-registry-private-configuration\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334637    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-registry-tls\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334685    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0882592c-76ee-4747-8c81-e62c4fe88a76-cert\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334711    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-trusted-ca\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334734    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-installation-pull-secrets\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334759    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9j8\" (UniqueName: \"kubernetes.io/projected/0882592c-76ee-4747-8c81-e62c4fe88a76-kube-api-access-8n9j8\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334784    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-registry-certificates\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334827    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpwq\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-kube-api-access-wbpwq\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334856    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23526c30-5d35-42a2-a14e-3525b1dde115-ca-trust-extracted\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.334890    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-bound-sa-token\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.336180    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-registry-certificates\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.336738    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23526c30-5d35-42a2-a14e-3525b1dde115-ca-trust-extracted\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.336789 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.336738    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23526c30-5d35-42a2-a14e-3525b1dde115-trusted-ca\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.337850 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.337655    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:10.338555 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.338458    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-image-registry-private-configuration\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.338555 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.338511    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23526c30-5d35-42a2-a14e-3525b1dde115-installation-pull-secrets\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.339134 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.339115    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-registry-tls\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.340097 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.340073    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0882592c-76ee-4747-8c81-e62c4fe88a76-cert\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.416610 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.415749    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpwq\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-kube-api-access-wbpwq\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.416821 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.416738    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23526c30-5d35-42a2-a14e-3525b1dde115-bound-sa-token\") pod \"image-registry-6cd75cdd8-lfzhm\" (UID: \"23526c30-5d35-42a2-a14e-3525b1dde115\") " pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.417062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.417009    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9j8\" (UniqueName: \"kubernetes.io/projected/0882592c-76ee-4747-8c81-e62c4fe88a76-kube-api-access-8n9j8\") pod \"ingress-canary-vzkxs\" (UID: \"0882592c-76ee-4747-8c81-e62c4fe88a76\") " pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.427462 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.427443    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzkxs"
Oct 30 21:50:10.431938 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.431920    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:10.628919 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.627123    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"]
Oct 30 21:50:10.631625 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.631600    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e767adc_6d0b_4e96_8b5b_9c5384fc5a07.slice/crio-074e6bb69e9508bfc7321eeee7e2b9f487b95d491a94c0f14b15b9ec0444a2e7 WatchSource:0}: Error finding container 074e6bb69e9508bfc7321eeee7e2b9f487b95d491a94c0f14b15b9ec0444a2e7: Status 404 returned error can't find the container with id 074e6bb69e9508bfc7321eeee7e2b9f487b95d491a94c0f14b15b9ec0444a2e7
Oct 30 21:50:10.675596 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.675574    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s7g72"]
Oct 30 21:50:10.678344 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.678321    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d01ed3_19c8_43e9_802d_da4f6a080e69.slice/crio-f31a6dd1cebe292a8a0fb21a88d7043243c21d17664036498cb95be0b45c6ab2 WatchSource:0}: Error finding container f31a6dd1cebe292a8a0fb21a88d7043243c21d17664036498cb95be0b45c6ab2: Status 404 returned error can't find the container with id f31a6dd1cebe292a8a0fb21a88d7043243c21d17664036498cb95be0b45c6ab2
Oct 30 21:50:10.695940 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.695923    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:10.697893 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.697867    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293391db_d36c_4eec_b5de_b6adb6cbb223.slice/crio-0bbec6e1da0aeb9ef995867ef03467685ec8109533065bed27c6455ef530aa4c WatchSource:0}: Error finding container 0bbec6e1da0aeb9ef995867ef03467685ec8109533065bed27c6455ef530aa4c: Status 404 returned error can't find the container with id 0bbec6e1da0aeb9ef995867ef03467685ec8109533065bed27c6455ef530aa4c
Oct 30 21:50:10.729745 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.729446    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5hv"]
Oct 30 21:50:10.731791 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.731772    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0e05f2_70bc_47c8_9a0d_93ff7223dcf6.slice/crio-f5034722cfb73a5959d66bb6c21da00ebb7691346d65e9d9bd0cfb601e562d56 WatchSource:0}: Error finding container f5034722cfb73a5959d66bb6c21da00ebb7691346d65e9d9bd0cfb601e562d56: Status 404 returned error can't find the container with id f5034722cfb73a5959d66bb6c21da00ebb7691346d65e9d9bd0cfb601e562d56
Oct 30 21:50:10.752950 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.752882    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vj5bd"]
Oct 30 21:50:10.758188 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.758163    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38de308_9d4a_46e7_80f9_59e5a61a2fdc.slice/crio-fa59cbe9825323f5b0fed384910656687b2d11a28f2656b985892fbf810dade7 WatchSource:0}: Error finding container fa59cbe9825323f5b0fed384910656687b2d11a28f2656b985892fbf810dade7: Status 404 returned error can't find the container with id fa59cbe9825323f5b0fed384910656687b2d11a28f2656b985892fbf810dade7
Oct 30 21:50:10.800474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.800445    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzkxs"]
Oct 30 21:50:10.803031 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.803011    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0882592c_76ee_4747_8c81_e62c4fe88a76.slice/crio-e8b9d7b1860f3fd56ea3ad92dc023a83dcb6cef75e5b1166be29a15cf15e7972 WatchSource:0}: Error finding container e8b9d7b1860f3fd56ea3ad92dc023a83dcb6cef75e5b1166be29a15cf15e7972: Status 404 returned error can't find the container with id e8b9d7b1860f3fd56ea3ad92dc023a83dcb6cef75e5b1166be29a15cf15e7972
Oct 30 21:50:10.812461 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:10.812439    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd75cdd8-lfzhm"]
Oct 30 21:50:10.814858 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:10.814836    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23526c30_5d35_42a2_a14e_3525b1dde115.slice/crio-9d26e2a1190f605f6744064f73d232c1b8eefba1849be5b6c4ab3da29a209fe9 WatchSource:0}: Error finding container 9d26e2a1190f605f6744064f73d232c1b8eefba1849be5b6c4ab3da29a209fe9: Status 404 returned error can't find the container with id 9d26e2a1190f605f6744064f73d232c1b8eefba1849be5b6c4ab3da29a209fe9
Oct 30 21:50:11.552416 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.552375    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm" event={"ID":"23526c30-5d35-42a2-a14e-3525b1dde115","Type":"ContainerStarted","Data":"e7b5622ecf954369ea3bec2a923e2d7553d2c810b3df4f1d7310c0a5cce780ca"}
Oct 30 21:50:11.552873 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.552443    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm" event={"ID":"23526c30-5d35-42a2-a14e-3525b1dde115","Type":"ContainerStarted","Data":"9d26e2a1190f605f6744064f73d232c1b8eefba1849be5b6c4ab3da29a209fe9"}
Oct 30 21:50:11.553201 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.553137    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:11.554883 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.554804    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vj5bd" event={"ID":"c38de308-9d4a-46e7-80f9-59e5a61a2fdc","Type":"ContainerStarted","Data":"fa59cbe9825323f5b0fed384910656687b2d11a28f2656b985892fbf810dade7"}
Oct 30 21:50:11.556682 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.556562    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5hv" event={"ID":"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6","Type":"ContainerStarted","Data":"745268083c687a9de005e100e42def63214a40b256b07a5c5b309408c801103e"}
Oct 30 21:50:11.556682 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.556611    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5hv" event={"ID":"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6","Type":"ContainerStarted","Data":"f5034722cfb73a5959d66bb6c21da00ebb7691346d65e9d9bd0cfb601e562d56"}
Oct 30 21:50:11.560114 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.558116    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f958b5dd-rrgpw" event={"ID":"293391db-d36c-4eec-b5de-b6adb6cbb223","Type":"ContainerStarted","Data":"0bbec6e1da0aeb9ef995867ef03467685ec8109533065bed27c6455ef530aa4c"}
Oct 30 21:50:11.560114 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.559503    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzkxs" event={"ID":"0882592c-76ee-4747-8c81-e62c4fe88a76","Type":"ContainerStarted","Data":"e8b9d7b1860f3fd56ea3ad92dc023a83dcb6cef75e5b1166be29a15cf15e7972"}
Oct 30 21:50:11.561845 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.561823    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s7g72" event={"ID":"80d01ed3-19c8-43e9-802d-da4f6a080e69","Type":"ContainerStarted","Data":"f31a6dd1cebe292a8a0fb21a88d7043243c21d17664036498cb95be0b45c6ab2"}
Oct 30 21:50:11.563626 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.563604    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt" event={"ID":"8e767adc-6d0b-4e96-8b5b-9c5384fc5a07","Type":"ContainerStarted","Data":"074e6bb69e9508bfc7321eeee7e2b9f487b95d491a94c0f14b15b9ec0444a2e7"}
Oct 30 21:50:11.695492 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:11.695432    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm" podStartSLOduration=1.695415581 podStartE2EDuration="1.695415581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:11.695030771 +0000 UTC m=+14.337624561" watchObservedRunningTime="2026-10-30 21:50:11.695415581 +0000 UTC m=+14.338009371"
Oct 30 21:50:12.570840 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:12.570794    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5hv" event={"ID":"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6","Type":"ContainerStarted","Data":"1d221f606939ab0dc2687efc4354654f98d8d0413f8977b72825333bfc70ac73"}
Oct 30 21:50:12.578582 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:12.578555    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt" event={"ID":"8e767adc-6d0b-4e96-8b5b-9c5384fc5a07","Type":"ContainerStarted","Data":"0032942715b6bc3aae6bd9f84a9371e60e9bc270192e73e5a307b5f4b018c1d3"}
Oct 30 21:50:12.578719 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:12.578705    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:12.585089 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:12.585064    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt"
Oct 30 21:50:12.698874 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:12.698819    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7nsmt" podStartSLOduration=1.942712137 podStartE2EDuration="3.69880252s" totalImagesPullingTime="1.756090383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:12.698515459 +0000 UTC m=+15.341109251" watchObservedRunningTime="2026-10-30 21:50:12.69880252 +0000 UTC m=+15.341396310"
Oct 30 21:50:14.144879 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.144855    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"]
Oct 30 21:50:14.149549 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.149528    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.204077 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.203949    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:50:14.204077 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.204001    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:50:14.204077 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.204050    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l4wxm\""
Oct 30 21:50:14.255798 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.254971    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:50:14.255798 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.255040    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:50:14.255798 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.254971    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:50:14.255798 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.255696    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"]
Oct 30 21:50:14.263599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.263404    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9c1409d-00e4-4c0b-8115-c095a39b58fe-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.263599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.263431    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.263599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.263460    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhk9\" (UniqueName: \"kubernetes.io/projected/c9c1409d-00e4-4c0b-8115-c095a39b58fe-kube-api-access-zfhk9\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.263599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.263519    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.364483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.364451    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.364626 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.364493    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9c1409d-00e4-4c0b-8115-c095a39b58fe-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.364626 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.364514    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.364626 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.364539    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhk9\" (UniqueName: \"kubernetes.io/projected/c9c1409d-00e4-4c0b-8115-c095a39b58fe-kube-api-access-zfhk9\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.365808 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.365782    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9c1409d-00e4-4c0b-8115-c095a39b58fe-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.368080 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.368054    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.368508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.368484    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9c1409d-00e4-4c0b-8115-c095a39b58fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.437206 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.437176    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhk9\" (UniqueName: \"kubernetes.io/projected/c9c1409d-00e4-4c0b-8115-c095a39b58fe-kube-api-access-zfhk9\") pod \"prometheus-operator-9cd89b74c-7mhkb\" (UID: \"c9c1409d-00e4-4c0b-8115-c095a39b58fe\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.471564 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.471535    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"
Oct 30 21:50:14.593334 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.593298    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vj5bd" event={"ID":"c38de308-9d4a-46e7-80f9-59e5a61a2fdc","Type":"ContainerStarted","Data":"3916e4648cb221d0738872be68ab3412e1f88185d298b6716660f2ba9ceb3e85"}
Oct 30 21:50:14.593456 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.593355    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vj5bd" event={"ID":"c38de308-9d4a-46e7-80f9-59e5a61a2fdc","Type":"ContainerStarted","Data":"385a11f1c04448233daa722c996e266fe1fb00ff899acfcb85d3fe8920fb45f8"}
Oct 30 21:50:14.593456 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.593406    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:14.720093 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.720029    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vj5bd" podStartSLOduration=2.474569849 podStartE2EDuration="5.720010797s" totalImagesPullingTime="3.245440948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.719421043 +0000 UTC m=+17.362014831" watchObservedRunningTime="2026-10-30 21:50:14.720010797 +0000 UTC m=+17.362604586"
Oct 30 21:50:14.786501 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.786473    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:50:14.790565 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.790543    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.868819 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.868789    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.868819 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.868832    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.868872    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.868905    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869087 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.869055    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869128 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.869096    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869166 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.869133    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.869201 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.869162    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfns\" (UniqueName: \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.898609 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.898038    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:50:14.898609 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.898568    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:50:14.925712 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:14.925684    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c1409d_00e4_4c0b_8115_c095a39b58fe.slice/crio-507bafc53bd859ba49621318f0afb40eaea58736893147c0ee547c19a46f7647 WatchSource:0}: Error finding container 507bafc53bd859ba49621318f0afb40eaea58736893147c0ee547c19a46f7647: Status 404 returned error can't find the container with id 507bafc53bd859ba49621318f0afb40eaea58736893147c0ee547c19a46f7647
Oct 30 21:50:14.945908 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.945882    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb"]
Oct 30 21:50:14.969616 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969591    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969633    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969649    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969694    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969721    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfns\" (UniqueName: \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969941 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969750    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.969941 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969921    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.970034 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.969970    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.971000 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.970953    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.971567 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.971228    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.971567 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.971520    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.971698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.971632    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.971878 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.971850    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.976748 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.976722    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:14.978336 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:14.978305    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:15.044627 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.044599    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfns\" (UniqueName: \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\") pod \"console-56888d4f6-tsl2m\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") " pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:15.105346 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.105249    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:15.501657 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.501633    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:50:15.503101 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:15.503074    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3309120f_1930_47ae_b84b_19b8dfba5534.slice/crio-64a7b0d2254831c9c09df6fdea9a3fabfba36854e29df69e10670ed6b11f457a WatchSource:0}: Error finding container 64a7b0d2254831c9c09df6fdea9a3fabfba36854e29df69e10670ed6b11f457a: Status 404 returned error can't find the container with id 64a7b0d2254831c9c09df6fdea9a3fabfba36854e29df69e10670ed6b11f457a
Oct 30 21:50:15.600218 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.600173    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5hv" event={"ID":"5a0e05f2-70bc-47c8-9a0d-93ff7223dcf6","Type":"ContainerStarted","Data":"7d747d8bcab7463fadebc774cc2fec26c6726542189f71a8c54a1beefda45f51"}
Oct 30 21:50:15.602807 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.602776    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f958b5dd-rrgpw" event={"ID":"293391db-d36c-4eec-b5de-b6adb6cbb223","Type":"ContainerStarted","Data":"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"}
Oct 30 21:50:15.605313 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.605258    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzkxs" event={"ID":"0882592c-76ee-4747-8c81-e62c4fe88a76","Type":"ContainerStarted","Data":"63e0487ee2801e0796f9fc99191c93e68fd7f24474ee6891b5d8c31c6f572ae2"}
Oct 30 21:50:15.607506 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.607425    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56888d4f6-tsl2m" event={"ID":"3309120f-1930-47ae-b84b-19b8dfba5534","Type":"ContainerStarted","Data":"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"}
Oct 30 21:50:15.607506 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.607464    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56888d4f6-tsl2m" event={"ID":"3309120f-1930-47ae-b84b-19b8dfba5534","Type":"ContainerStarted","Data":"64a7b0d2254831c9c09df6fdea9a3fabfba36854e29df69e10670ed6b11f457a"}
Oct 30 21:50:15.609468 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.609378    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb" event={"ID":"c9c1409d-00e4-4c0b-8115-c095a39b58fe","Type":"ContainerStarted","Data":"507bafc53bd859ba49621318f0afb40eaea58736893147c0ee547c19a46f7647"}
Oct 30 21:50:15.729959 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.729901    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nm5hv" podStartSLOduration=2.70117219 podStartE2EDuration="6.729888329s" totalImagesPullingTime="4.028716139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.729201015 +0000 UTC m=+18.371794807" watchObservedRunningTime="2026-10-30 21:50:15.729888329 +0000 UTC m=+18.372482115"
Oct 30 21:50:15.866081 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.865974    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vzkxs" podStartSLOduration=1.3104794370000001 podStartE2EDuration="5.865954848s" totalImagesPullingTime="4.555475411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.865392653 +0000 UTC m=+18.507986442" watchObservedRunningTime="2026-10-30 21:50:15.865954848 +0000 UTC m=+18.508548635"
Oct 30 21:50:15.996029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:15.995967    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56888d4f6-tsl2m" podStartSLOduration=1.995947532 podStartE2EDuration="1.995947532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.995323297 +0000 UTC m=+18.637917089" watchObservedRunningTime="2026-10-30 21:50:15.995947532 +0000 UTC m=+18.638541322"
Oct 30 21:50:17.618849 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:17.618812    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb" event={"ID":"c9c1409d-00e4-4c0b-8115-c095a39b58fe","Type":"ContainerStarted","Data":"fff97ba6e9e1c8f85e80cebf1f2dd9ac655a474d55cf102b226f1fdb43ecbcb2"}
Oct 30 21:50:17.619277 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:17.618869    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb" event={"ID":"c9c1409d-00e4-4c0b-8115-c095a39b58fe","Type":"ContainerStarted","Data":"dff038b9200a4f5217b0ef62f46dd3684689b44ec8981ac761dd289399130727"}
Oct 30 21:50:17.745585 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:17.745530    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68f958b5dd-rrgpw" podStartSLOduration=4.080909215 podStartE2EDuration="8.745512499s" totalImagesPullingTime="4.664603284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:16.129106125 +0000 UTC m=+18.771699917" watchObservedRunningTime="2026-10-30 21:50:17.745512499 +0000 UTC m=+20.388106288"
Oct 30 21:50:17.746081 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:17.746045    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mhkb" podStartSLOduration=2.004867513 podStartE2EDuration="3.746034701s" totalImagesPullingTime="1.741167188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.744879824 +0000 UTC m=+20.387473614" watchObservedRunningTime="2026-10-30 21:50:17.746034701 +0000 UTC m=+20.388628489"
Oct 30 21:50:20.279745 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.279705    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:20.280213 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.279824    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:20.284676 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.284637    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:20.634619 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.634547    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:20.742261 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.742227    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2rb6f"]
Oct 30 21:50:20.744503 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.744485    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816072 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816035    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-textfile\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816234 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816088    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-metrics-client-ca\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816234 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816211    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816323 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816254    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816323 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816290    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-wtmp\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816423 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816327    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-root\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816423 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816372    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-tls\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816515 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816446    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-sys\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.816566 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.816516    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvzh\" (UniqueName: \"kubernetes.io/projected/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-kube-api-access-pmvzh\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.841262 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.841238    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:50:20.841262 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.841261    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:50:20.841464 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.841293    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:50:20.841464 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.841327    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5k6j\""
Oct 30 21:50:20.917921 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.917843    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-metrics-client-ca\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.917921 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.917894    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918120 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918037    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918120 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918093    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-wtmp\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918223 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918126    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-root\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918223 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918150    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-tls\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918223 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918177    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-sys\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918371 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918226    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvzh\" (UniqueName: \"kubernetes.io/projected/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-kube-api-access-pmvzh\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918371 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918229    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-root\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918371 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918275    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-wtmp\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918371 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918289    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-textfile\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918562 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918479    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-sys\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918562 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918527    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918880 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918862    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-metrics-client-ca\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.918993 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.918977    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-textfile\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.920967 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.920945    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.921135 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.921115    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-node-exporter-tls\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:20.989288 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:20.989259    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvzh\" (UniqueName: \"kubernetes.io/projected/ab8ebc85-ba38-41f1-a38e-80bdcffc6005-kube-api-access-pmvzh\") pod \"node-exporter-2rb6f\" (UID: \"ab8ebc85-ba38-41f1-a38e-80bdcffc6005\") " pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:21.057483 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:21.057455    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rb6f"
Oct 30 21:50:21.069209 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:21.069183    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8ebc85_ba38_41f1_a38e_80bdcffc6005.slice/crio-1875b36185257e9f330690be95fbf487844dcfb56a81b208059d2414ec593f67 WatchSource:0}: Error finding container 1875b36185257e9f330690be95fbf487844dcfb56a81b208059d2414ec593f67: Status 404 returned error can't find the container with id 1875b36185257e9f330690be95fbf487844dcfb56a81b208059d2414ec593f67
Oct 30 21:50:21.635867 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:21.635825    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rb6f" event={"ID":"ab8ebc85-ba38-41f1-a38e-80bdcffc6005","Type":"ContainerStarted","Data":"1875b36185257e9f330690be95fbf487844dcfb56a81b208059d2414ec593f67"}
Oct 30 21:50:22.642109 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:22.642076    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rb6f" event={"ID":"ab8ebc85-ba38-41f1-a38e-80bdcffc6005","Type":"ContainerStarted","Data":"3af8184af040082a980e8a2bb3da72fc07e4626caefc3ea2865293579352a3e6"}
Oct 30 21:50:23.609900 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:50:23.609864    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:50:23.649484 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:23.649453    4295 generic.go:356] "Generic (PLEG): container finished" podID="ab8ebc85-ba38-41f1-a38e-80bdcffc6005" containerID="3af8184af040082a980e8a2bb3da72fc07e4626caefc3ea2865293579352a3e6" exitCode=0
Oct 30 21:50:23.649847 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:23.649545    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rb6f" event={"ID":"ab8ebc85-ba38-41f1-a38e-80bdcffc6005","Type":"ContainerDied","Data":"3af8184af040082a980e8a2bb3da72fc07e4626caefc3ea2865293579352a3e6"}
Oct 30 21:50:24.655578 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:24.655542    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rb6f" event={"ID":"ab8ebc85-ba38-41f1-a38e-80bdcffc6005","Type":"ContainerStarted","Data":"8d91288e93e3ba73a4528be1e815e6c8f5281a7800b99b1661c575330cecd489"}
Oct 30 21:50:24.655986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:24.655626    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rb6f" event={"ID":"ab8ebc85-ba38-41f1-a38e-80bdcffc6005","Type":"ContainerStarted","Data":"26c7e80117a48df67251fe1867a45b96ecb05de4426fa77b4fbfd6566238850a"}
Oct 30 21:50:24.778013 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:24.777966    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2rb6f" podStartSLOduration=3.205863649 podStartE2EDuration="4.777951497s" totalImagesPullingTime="1.493399603s" totalInitContainerRuntime="78.688245ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:24.777334645 +0000 UTC m=+27.419928449" watchObservedRunningTime="2026-10-30 21:50:24.777951497 +0000 UTC m=+27.420545285"
Oct 30 21:50:25.105682 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.105642    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:25.105853 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.105718    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:25.111373 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.111347    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:25.614777 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.614749    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vj5bd"
Oct 30 21:50:25.663944 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.663912    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:50:25.899941 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:25.899869    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:28.450876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:28.450841    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nj82c"
Oct 30 21:50:31.682529 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:31.682492    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s7g72" event={"ID":"80d01ed3-19c8-43e9-802d-da4f6a080e69","Type":"ContainerStarted","Data":"41b56cb9d47c2fa22e0b09016e7b45f1496cbc176b46875818cc3b1cd0180cd8"}
Oct 30 21:50:31.683006 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:31.682875    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:31.683942 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:31.683911    4295 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-s7g72 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" start-of-body=
Oct 30 21:50:31.684053 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:31.683970    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-s7g72" podUID="80d01ed3-19c8-43e9-802d-da4f6a080e69" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused"
Oct 30 21:50:32.656050 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.655990    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-s7g72" podStartSLOduration=2.810224523 podStartE2EDuration="23.655971492s" totalImagesPullingTime="20.845746969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:31.811382681 +0000 UTC m=+34.453976472" watchObservedRunningTime="2026-10-30 21:50:32.655971492 +0000 UTC m=+35.298565319"
Oct 30 21:50:32.656911 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.656885    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:50:32.685522 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.685488    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.700940 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.700912    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-s7g72"
Oct 30 21:50:32.721330 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.721297    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:50:32.821085 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821053    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821255 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821209    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821337 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821302    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821422 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821356    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821422 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821378    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gztq\" (UniqueName: \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821422 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821414    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821466    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.821599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.821498    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924349    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924416    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924443    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924491    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924518    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gztq\" (UniqueName: \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924557    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.924876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.924583    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.925345 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.925159    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.925407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.925371    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.926195 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.926163    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.926721 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.926353    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.926721 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.926651    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.927451 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.927427    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.929970 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.929942    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.930080 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.930018    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:32.997648 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:32.997615    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gztq\" (UniqueName: \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\") pod \"console-c4d789b9f-xzffh\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") " pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:33.021389 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.021311    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:33.327714 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.327687    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6qwzk"
Oct 30 21:50:33.437811 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.437768    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:50:33.442031 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:50:33.442000    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74a1b3d_1623_4472_b361_5d72c4e14750.slice/crio-3864605ca3b0a8c1fbb14f269b04fad39bf8d08531fe43ba3b18fdc71cc3ee24 WatchSource:0}: Error finding container 3864605ca3b0a8c1fbb14f269b04fad39bf8d08531fe43ba3b18fdc71cc3ee24: Status 404 returned error can't find the container with id 3864605ca3b0a8c1fbb14f269b04fad39bf8d08531fe43ba3b18fdc71cc3ee24
Oct 30 21:50:33.588741 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.587355    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cd75cdd8-lfzhm"
Oct 30 21:50:33.693124 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.693086    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4d789b9f-xzffh" event={"ID":"d74a1b3d-1623-4472-b361-5d72c4e14750","Type":"ContainerStarted","Data":"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"}
Oct 30 21:50:33.693583 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.693144    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4d789b9f-xzffh" event={"ID":"d74a1b3d-1623-4472-b361-5d72c4e14750","Type":"ContainerStarted","Data":"3864605ca3b0a8c1fbb14f269b04fad39bf8d08531fe43ba3b18fdc71cc3ee24"}
Oct 30 21:50:33.833804 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:33.833741    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c4d789b9f-xzffh" podStartSLOduration=1.833724517 podStartE2EDuration="1.833724517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:33.833304317 +0000 UTC m=+36.475898110" watchObservedRunningTime="2026-10-30 21:50:33.833724517 +0000 UTC m=+36.476318307"
Oct 30 21:50:43.022325 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:43.022237    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:43.022805 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:43.022526    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:43.027137 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:43.027116    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:43.737486 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:43.737455    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:50:43.910797 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:43.910764    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:50:50.923156 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:50.923093    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68f958b5dd-rrgpw" podUID="293391db-d36c-4eec-b5de-b6adb6cbb223" containerName="console" containerID="cri-o://0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902" gracePeriod=15
Oct 30 21:50:51.200894 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.200870    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:51.277853 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.277824    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.277985 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.277880    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.277985 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.277908    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\" (UniqueName: \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.277985 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.277968    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.278194 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.277997    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\" (UniqueName: \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.278194 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278031    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.278194 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278061    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\") pod \"293391db-d36c-4eec-b5de-b6adb6cbb223\" (UID: \"293391db-d36c-4eec-b5de-b6adb6cbb223\") "
Oct 30 21:50:51.278353 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278300    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:51.278467 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278437    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.278580 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278503    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.278580 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.278542    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.280273 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.280233    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:51.280273 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.280258    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:51.280424 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.280303    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs" pod "293391db-d36c-4eec-b5de-b6adb6cbb223" (UID: "293391db-d36c-4eec-b5de-b6adb6cbb223"). InnerVolumeSpecName "kube-api-access-dgjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:51.379466 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379434    4295 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-oauth-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379466 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379458    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgjgs\" (UniqueName: \"kubernetes.io/projected/293391db-d36c-4eec-b5de-b6adb6cbb223-kube-api-access-dgjgs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379466 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379470    4295 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379687 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379478    4295 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293391db-d36c-4eec-b5de-b6adb6cbb223-tmp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379687 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379487    4295 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379687 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379496    4295 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/293391db-d36c-4eec-b5de-b6adb6cbb223-console-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.379687 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.379506    4295 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/293391db-d36c-4eec-b5de-b6adb6cbb223-console-oauth-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.762353 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.762280    4295 generic.go:356] "Generic (PLEG): container finished" podID="293391db-d36c-4eec-b5de-b6adb6cbb223" containerID="0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902" exitCode=0
Oct 30 21:50:51.762353 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.762330    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f958b5dd-rrgpw"
Oct 30 21:50:51.762525 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.762368    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f958b5dd-rrgpw" event={"ID":"293391db-d36c-4eec-b5de-b6adb6cbb223","Type":"ContainerDied","Data":"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"}
Oct 30 21:50:51.762525 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.762434    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f958b5dd-rrgpw" event={"ID":"293391db-d36c-4eec-b5de-b6adb6cbb223","Type":"ContainerDied","Data":"0bbec6e1da0aeb9ef995867ef03467685ec8109533065bed27c6455ef530aa4c"}
Oct 30 21:50:51.762525 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.762459    4295 scope.go:122] "RemoveContainer" containerID="0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"
Oct 30 21:50:51.770269 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.770251    4295 scope.go:122] "RemoveContainer" containerID="0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"
Oct 30 21:50:51.770550 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:50:51.770532    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902\": container with ID starting with 0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902 not found: ID does not exist" containerID="0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"
Oct 30 21:50:51.770620 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.770558    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902"} err="failed to get container status \"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902\": rpc error: code = NotFound desc = could not find container \"0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902\": container with ID starting with 0a2f518d6a4adcab3f55a370136463fd2028f88232bbb021fada7115a6a97902 not found: ID does not exist"
Oct 30 21:50:51.930339 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.930302    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:51.980708 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:51.980679    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f958b5dd-rrgpw"]
Oct 30 21:50:53.490324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:50:53.490294    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293391db-d36c-4eec-b5de-b6adb6cbb223" path="/var/lib/kubelet/pods/293391db-d36c-4eec-b5de-b6adb6cbb223/volumes"
Oct 30 21:50:53.606961 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:50:53.606926    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:51:08.931025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:08.930959    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56888d4f6-tsl2m" podUID="3309120f-1930-47ae-b84b-19b8dfba5534" containerName="console" containerID="cri-o://3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e" gracePeriod=15
Oct 30 21:51:09.201464 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.201407    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:51:09.308469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308438    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308471    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308506    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308537    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308565    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308600    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308638    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\" (UniqueName: \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308998 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308689    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\") pod \"3309120f-1930-47ae-b84b-19b8dfba5534\" (UID: \"3309120f-1930-47ae-b84b-19b8dfba5534\") "
Oct 30 21:51:09.308998 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308806    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:09.308998 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308981    4295 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3309120f-1930-47ae-b84b-19b8dfba5534-tmp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.309163 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.308994    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:09.309163 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.309125    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:09.309265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.309234    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:09.309265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.309230    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:09.310729 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.310707    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:09.310937 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.310922    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:09.311029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.311014    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns" pod "3309120f-1930-47ae-b84b-19b8dfba5534" (UID: "3309120f-1930-47ae-b84b-19b8dfba5534"). InnerVolumeSpecName "kube-api-access-klfns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:09.409304 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409269    4295 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-oauth-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409304 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409297    4295 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409304 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409308    4295 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3309120f-1930-47ae-b84b-19b8dfba5534-console-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409317    4295 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-console-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409325    4295 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-trusted-ca-bundle\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409334    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfns\" (UniqueName: \"kubernetes.io/projected/3309120f-1930-47ae-b84b-19b8dfba5534-kube-api-access-klfns\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.409576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.409343    4295 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3309120f-1930-47ae-b84b-19b8dfba5534-oauth-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:09.724524 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724492    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b56d5f68c-zzh6l"]
Oct 30 21:51:09.724767 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724754    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3309120f-1930-47ae-b84b-19b8dfba5534" containerName="console"
Oct 30 21:51:09.724817 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724768    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3309120f-1930-47ae-b84b-19b8dfba5534" containerName="console"
Oct 30 21:51:09.724817 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724791    4295 cpu_manager.go:414] "containerMap: removing container" podUID="293391db-d36c-4eec-b5de-b6adb6cbb223" containerName="console"
Oct 30 21:51:09.724817 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724796    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293391db-d36c-4eec-b5de-b6adb6cbb223" containerName="console"
Oct 30 21:51:09.724909 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724836    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3309120f-1930-47ae-b84b-19b8dfba5534" containerName="console"
Oct 30 21:51:09.724909 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.724848    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="293391db-d36c-4eec-b5de-b6adb6cbb223" containerName="console"
Oct 30 21:51:09.757584 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.757536    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.810903 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.810870    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811023 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.810907    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-oauth-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811023 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.810940    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7v66\" (UniqueName: \"kubernetes.io/projected/e60776b8-d447-4841-9482-faa6ff7f6e83-kube-api-access-r7v66\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811093 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.811041    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-oauth-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811093 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.811072    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-console-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811201 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.811096    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60776b8-d447-4841-9482-faa6ff7f6e83-tmp\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811201 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.811177    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-trusted-ca-bundle\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.811260 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.811200    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-service-ca\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.827672 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.827632    4295 generic.go:356] "Generic (PLEG): container finished" podID="3309120f-1930-47ae-b84b-19b8dfba5534" containerID="3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e" exitCode=0
Oct 30 21:51:09.827766 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.827691    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56888d4f6-tsl2m" event={"ID":"3309120f-1930-47ae-b84b-19b8dfba5534","Type":"ContainerDied","Data":"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"}
Oct 30 21:51:09.827766 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.827719    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56888d4f6-tsl2m"
Oct 30 21:51:09.827766 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.827741    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56888d4f6-tsl2m" event={"ID":"3309120f-1930-47ae-b84b-19b8dfba5534","Type":"ContainerDied","Data":"64a7b0d2254831c9c09df6fdea9a3fabfba36854e29df69e10670ed6b11f457a"}
Oct 30 21:51:09.827861 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.827768    4295 scope.go:122] "RemoveContainer" containerID="3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"
Oct 30 21:51:09.831698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.831655    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b56d5f68c-zzh6l"]
Oct 30 21:51:09.836075 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.836061    4295 scope.go:122] "RemoveContainer" containerID="3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"
Oct 30 21:51:09.836332 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:51:09.836314    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e\": container with ID starting with 3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e not found: ID does not exist" containerID="3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"
Oct 30 21:51:09.836382 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.836341    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e"} err="failed to get container status \"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e\": rpc error: code = NotFound desc = could not find container \"3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e\": container with ID starting with 3b1050d112989ae51a5abb878806c421c1f51c455c253c6d6f6f16a8e0f8b29e not found: ID does not exist"
Oct 30 21:51:09.911855 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911828    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-oauth-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.911971 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911869    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-console-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.911971 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911901    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60776b8-d447-4841-9482-faa6ff7f6e83-tmp\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.911971 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911928    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-trusted-ca-bundle\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.911971 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911959    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-service-ca\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912173 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.911986    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912173 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912014    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-oauth-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912173 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912054    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7v66\" (UniqueName: \"kubernetes.io/projected/e60776b8-d447-4841-9482-faa6ff7f6e83-kube-api-access-r7v66\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912336 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912315    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60776b8-d447-4841-9482-faa6ff7f6e83-tmp\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912706 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912656    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-console-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912820 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912737    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-service-ca\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.912975 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.912954    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-trusted-ca-bundle\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.913035 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.913014    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e60776b8-d447-4841-9482-faa6ff7f6e83-oauth-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.914417 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.914401    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-oauth-config\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.914554 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.914535    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e60776b8-d447-4841-9482-faa6ff7f6e83-console-serving-cert\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.975213 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.975190    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7v66\" (UniqueName: \"kubernetes.io/projected/e60776b8-d447-4841-9482-faa6ff7f6e83-kube-api-access-r7v66\") pod \"console-6b56d5f68c-zzh6l\" (UID: \"e60776b8-d447-4841-9482-faa6ff7f6e83\") " pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:09.990890 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:09.990869    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:51:10.002721 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.002704    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56888d4f6-tsl2m"]
Oct 30 21:51:10.079010 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.078936    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:10.484036 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.484004    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b56d5f68c-zzh6l"]
Oct 30 21:51:10.486806 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:51:10.486781    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60776b8_d447_4841_9482_faa6ff7f6e83.slice/crio-09dff8e1202c9d0c1c804c730384fa93284c796d5f55263ee24411cac6647a27 WatchSource:0}: Error finding container 09dff8e1202c9d0c1c804c730384fa93284c796d5f55263ee24411cac6647a27: Status 404 returned error can't find the container with id 09dff8e1202c9d0c1c804c730384fa93284c796d5f55263ee24411cac6647a27
Oct 30 21:51:10.833435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.833354    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b56d5f68c-zzh6l" event={"ID":"e60776b8-d447-4841-9482-faa6ff7f6e83","Type":"ContainerStarted","Data":"ee7346674595068ad0004934dfa1d5f479a86112343eb6eea7de46862848ab70"}
Oct 30 21:51:10.833435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.833397    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b56d5f68c-zzh6l" event={"ID":"e60776b8-d447-4841-9482-faa6ff7f6e83","Type":"ContainerStarted","Data":"09dff8e1202c9d0c1c804c730384fa93284c796d5f55263ee24411cac6647a27"}
Oct 30 21:51:10.956080 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:10.956034    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b56d5f68c-zzh6l" podStartSLOduration=1.956022836 podStartE2EDuration="1.956022836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:10.955483309 +0000 UTC m=+73.598077099" watchObservedRunningTime="2026-10-30 21:51:10.956022836 +0000 UTC m=+73.598616625"
Oct 30 21:51:11.490570 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:11.490539    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3309120f-1930-47ae-b84b-19b8dfba5534" path="/var/lib/kubelet/pods/3309120f-1930-47ae-b84b-19b8dfba5534/volumes"
Oct 30 21:51:20.079741 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:20.079704    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:20.080215 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:20.079755    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:20.084382 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:20.084363    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:20.868265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:20.868235    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b56d5f68c-zzh6l"
Oct 30 21:51:21.043036 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:21.042316    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:51:23.606598 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:51:23.606568    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:51:46.068021 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.067955    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c4d789b9f-xzffh" podUID="d74a1b3d-1623-4472-b361-5d72c4e14750" containerName="console" containerID="cri-o://d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb" gracePeriod=15
Oct 30 21:51:46.300297 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.300259    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:51:46.374140 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374070    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374140 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374105    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374140 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374132    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374345 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374265    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374345 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374313    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374366    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374380    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:46.374435 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374395    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374584 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374466    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\" (UniqueName: \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\") pod \"d74a1b3d-1623-4472-b361-5d72c4e14750\" (UID: \"d74a1b3d-1623-4472-b361-5d72c4e14750\") "
Oct 30 21:51:46.374584 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374489    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:46.374584 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374513    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:46.374764 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374731    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:46.374864 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374839    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:46.374968 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374952    4295 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-console-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.375027 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374975    4295 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-trusted-ca-bundle\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.375027 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.374990    4295 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74a1b3d-1623-4472-b361-5d72c4e14750-tmp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.375027 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.375005    4295 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-oauth-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.375027 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.375019    4295 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a1b3d-1623-4472-b361-5d72c4e14750-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.376419 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.376397    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:46.376517 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.376495    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:46.376716 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.376693    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq" pod "d74a1b3d-1623-4472-b361-5d72c4e14750" (UID: "d74a1b3d-1623-4472-b361-5d72c4e14750"). InnerVolumeSpecName "kube-api-access-2gztq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:46.476331 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.476297    4295 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-oauth-config\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.476331 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.476324    4295 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74a1b3d-1623-4472-b361-5d72c4e14750-console-serving-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.476331 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.476335    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gztq\" (UniqueName: \"kubernetes.io/projected/d74a1b3d-1623-4472-b361-5d72c4e14750-kube-api-access-2gztq\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:46.951156 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.951113    4295 generic.go:356] "Generic (PLEG): container finished" podID="d74a1b3d-1623-4472-b361-5d72c4e14750" containerID="d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb" exitCode=0
Oct 30 21:51:46.951298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.951165    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4d789b9f-xzffh"
Oct 30 21:51:46.951298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.951202    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4d789b9f-xzffh" event={"ID":"d74a1b3d-1623-4472-b361-5d72c4e14750","Type":"ContainerDied","Data":"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"}
Oct 30 21:51:46.951298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.951254    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4d789b9f-xzffh" event={"ID":"d74a1b3d-1623-4472-b361-5d72c4e14750","Type":"ContainerDied","Data":"3864605ca3b0a8c1fbb14f269b04fad39bf8d08531fe43ba3b18fdc71cc3ee24"}
Oct 30 21:51:46.951298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.951274    4295 scope.go:122] "RemoveContainer" containerID="d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"
Oct 30 21:51:46.959279 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.959263    4295 scope.go:122] "RemoveContainer" containerID="d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"
Oct 30 21:51:46.959540 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:51:46.959521    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb\": container with ID starting with d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb not found: ID does not exist" containerID="d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"
Oct 30 21:51:46.959589 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:46.959549    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb"} err="failed to get container status \"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb\": rpc error: code = NotFound desc = could not find container \"d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb\": container with ID starting with d15c6846343df9e3f74af4a886742b1100c3432cc6d05a5c920f9e1db192bbcb not found: ID does not exist"
Oct 30 21:51:47.119489 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:47.119454    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:51:47.169973 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:47.169939    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c4d789b9f-xzffh"]
Oct 30 21:51:47.490858 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:51:47.490828    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74a1b3d-1623-4472-b361-5d72c4e14750" path="/var/lib/kubelet/pods/d74a1b3d-1623-4472-b361-5d72c4e14750/volumes"
Oct 30 21:51:53.608354 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:51:53.608310    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:52:23.606555 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:52:23.606478    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:52:52.085913 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.085874    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"]
Oct 30 21:52:52.086377 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.086200    4295 cpu_manager.go:414] "containerMap: removing container" podUID="d74a1b3d-1623-4472-b361-5d72c4e14750" containerName="console"
Oct 30 21:52:52.086377 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.086213    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a1b3d-1623-4472-b361-5d72c4e14750" containerName="console"
Oct 30 21:52:52.086377 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.086270    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74a1b3d-1623-4472-b361-5d72c4e14750" containerName="console"
Oct 30 21:52:52.088029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.088014    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.139226 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.139199    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.139334 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.139255    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xqs\" (UniqueName: \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.139334 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.139277    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.192420 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.192392    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:52:52.192420 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.192391    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:52:52.192631 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.192609    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-r29x9\""
Oct 30 21:52:52.193541 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.193523    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"]
Oct 30 21:52:52.239876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.239854    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xqs\" (UniqueName: \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.240030 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.239883    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.240030 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.239906    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.240243 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.240226    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.240324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.240273    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.303413 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.303390    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xqs\" (UniqueName: \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.390090 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.390012    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:52:52.392062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.392042    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.400078 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.400058    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:52:52.441582 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441550    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.441750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441600    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.441750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441685    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.441750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441710    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.441750 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441730    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.442047 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441758    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxjt\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.442047 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.441870    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.449782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.449511    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:52:52.449782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.449563    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:52:52.449782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.449589    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6s5w5\""
Oct 30 21:52:52.449782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.449630    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:52:52.450616 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.450473    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:52:52.450616 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.450552    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:52:52.450616 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.450592    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:52:52.457469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.457398    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:52:52.543133 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543102    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543149    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543170    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543188    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543204    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543233    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxjt\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.543306 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.543263    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.544186 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.544158    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.545532 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.545504    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.545734 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.545714    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.545805 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.545738    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.545805 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.545722    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.606939 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.606917    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.607140 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.607123    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxjt\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\") pod \"istiod-openshift-gateway-7876bfcb74-6qbp2\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.701936 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.701917    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:52.772138 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:52.772024    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"]
Oct 30 21:52:52.775248 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:52:52.775221    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70778c2e_40f3_454d_9ae4_cdee57bba326.slice/crio-454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509 WatchSource:0}: Error finding container 454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509: Status 404 returned error can't find the container with id 454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509
Oct 30 21:52:53.072769 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:53.072689    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:52:53.075654 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:52:53.075630    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f09cce_c09e_494c_930c_08fc6d0535af.slice/crio-1240c09bd11384b17f6d347afa9c5305d7f825f4209df6adc184b3561c95396f WatchSource:0}: Error finding container 1240c09bd11384b17f6d347afa9c5305d7f825f4209df6adc184b3561c95396f: Status 404 returned error can't find the container with id 1240c09bd11384b17f6d347afa9c5305d7f825f4209df6adc184b3561c95396f
Oct 30 21:52:53.173936 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:53.173906    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" event={"ID":"e1f09cce-c09e-494c-930c-08fc6d0535af","Type":"ContainerStarted","Data":"1240c09bd11384b17f6d347afa9c5305d7f825f4209df6adc184b3561c95396f"}
Oct 30 21:52:53.175015 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:53.174995    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerStarted","Data":"454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509"}
Oct 30 21:52:53.607011 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:52:53.606978    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:52:56.921698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:56.921651    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:52:56.921948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:56.921731    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:52:57.191201 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:57.191166    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" event={"ID":"e1f09cce-c09e-494c-930c-08fc6d0535af","Type":"ContainerStarted","Data":"a058dd19d68c8bb6d0932fcb67901c50c1b5aa1f885f8066e5a7bfd3915c18a9"}
Oct 30 21:52:57.191400 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:57.191366    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:52:57.192990 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:57.192955    4295 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6qbp2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:52:57.193083 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:57.193016    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:52:57.402152 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:57.402099    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" podStartSLOduration=1.5587967150000002 podStartE2EDuration="5.402079067s" totalImagesPullingTime="3.843282352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:57.366679886 +0000 UTC m=+180.009273666" watchObservedRunningTime="2026-10-30 21:52:57.402079067 +0000 UTC m=+180.044672850"
Oct 30 21:52:58.195077 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:58.195043    4295 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6qbp2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:52:58.195523 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:52:58.195097    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:53:00.210226 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:00.210169    4295 generic.go:356] "Generic (PLEG): container finished" podID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerID="4e02f88daa2e74c73aee139152e8cd84d5532251dca0bb98f56580bf2233cb75" exitCode=0
Oct 30 21:53:00.210700 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:00.210260    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerDied","Data":"4e02f88daa2e74c73aee139152e8cd84d5532251dca0bb98f56580bf2233cb75"}
Oct 30 21:53:01.196114 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:01.196087    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:53:02.218628 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:02.218597    4295 generic.go:356] "Generic (PLEG): container finished" podID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerID="f7de17fdeb85afe36c729d9ff7d6789d51e108546dcbd36cae6c26d03b1c8f11" exitCode=0
Oct 30 21:53:02.219021 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:02.218639    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerDied","Data":"f7de17fdeb85afe36c729d9ff7d6789d51e108546dcbd36cae6c26d03b1c8f11"}
Oct 30 21:53:09.248530 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:09.248498    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerStarted","Data":"4b7a2723bf17c54ab0a93954186f6220a74831949d37641054d621482adcf98f"}
Oct 30 21:53:09.372090 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:09.372030    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" podStartSLOduration=3.139041483 podStartE2EDuration="17.372015345s" totalImagesPullingTime="14.232973862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:09.370677861 +0000 UTC m=+192.013271648" watchObservedRunningTime="2026-10-30 21:53:09.372015345 +0000 UTC m=+192.014609134"
Oct 30 21:53:10.254430 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:10.254396    4295 generic.go:356] "Generic (PLEG): container finished" podID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerID="4b7a2723bf17c54ab0a93954186f6220a74831949d37641054d621482adcf98f" exitCode=0
Oct 30 21:53:10.254836 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:10.254484    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerDied","Data":"4b7a2723bf17c54ab0a93954186f6220a74831949d37641054d621482adcf98f"}
Oct 30 21:53:11.373305 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.373283    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:53:11.504449 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.504413    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\" (UniqueName: \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\") pod \"70778c2e-40f3-454d-9ae4-cdee57bba326\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") "
Oct 30 21:53:11.504622 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.504471    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\") pod \"70778c2e-40f3-454d-9ae4-cdee57bba326\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") "
Oct 30 21:53:11.504622 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.504581    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\") pod \"70778c2e-40f3-454d-9ae4-cdee57bba326\" (UID: \"70778c2e-40f3-454d-9ae4-cdee57bba326\") "
Oct 30 21:53:11.505094 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.505063    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle" pod "70778c2e-40f3-454d-9ae4-cdee57bba326" (UID: "70778c2e-40f3-454d-9ae4-cdee57bba326"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:11.506767 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.506742    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs" pod "70778c2e-40f3-454d-9ae4-cdee57bba326" (UID: "70778c2e-40f3-454d-9ae4-cdee57bba326"). InnerVolumeSpecName "kube-api-access-r8xqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:11.509137 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.509084    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util" pod "70778c2e-40f3-454d-9ae4-cdee57bba326" (UID: "70778c2e-40f3-454d-9ae4-cdee57bba326"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:11.605508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.605450    4295 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-bundle\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:11.605508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.605499    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xqs\" (UniqueName: \"kubernetes.io/projected/70778c2e-40f3-454d-9ae4-cdee57bba326-kube-api-access-r8xqs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:11.605508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:11.605510    4295 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70778c2e-40f3-454d-9ae4-cdee57bba326-util\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:12.262778 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:12.262753    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4"
Oct 30 21:53:12.262937 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:12.262747    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71stck4" event={"ID":"70778c2e-40f3-454d-9ae4-cdee57bba326","Type":"ContainerDied","Data":"454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509"}
Oct 30 21:53:12.262937 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:12.262875    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454121b3a873f5ead836483cca04b84d3560706b573da086071e546d39ccc509"
Oct 30 21:53:23.607880 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:53:23.607844    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.816804    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"]
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817297    4295 cpu_manager.go:414] "containerMap: removing container" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="pull"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817318    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="pull"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817366    4295 cpu_manager.go:414] "containerMap: removing container" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="extract"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817373    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="extract"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817387    4295 cpu_manager.go:414] "containerMap: removing container" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="util"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817394    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="util"
Oct 30 21:53:26.819232 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.817507    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="70778c2e-40f3-454d-9ae4-cdee57bba326" containerName="extract"
Oct 30 21:53:26.822340 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.822322    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914177 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914144    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914183    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914203    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914292    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914321    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.914454 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.914350    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsl7\" (UniqueName: \"kubernetes.io/projected/81870999-5195-4f20-807d-c4adaf5c08bc-kube-api-access-hfsl7\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:26.937636 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937593    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:53:26.937797 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937606    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:53:26.937876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937834    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:53:26.937876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937833    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:53:26.937876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937855    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:53:26.937876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.937834    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4xvvf\""
Oct 30 21:53:26.942838 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:26.942815    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"]
Oct 30 21:53:27.014979 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.014945    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsl7\" (UniqueName: \"kubernetes.io/projected/81870999-5195-4f20-807d-c4adaf5c08bc-kube-api-access-hfsl7\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.014995    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.015026    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.015051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.015112    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015159 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.015150    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.015523 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.015492    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.017821 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.017800    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.017923 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.017843    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81870999-5195-4f20-807d-c4adaf5c08bc-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.017923 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.017843    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.018075 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.018057    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81870999-5195-4f20-807d-c4adaf5c08bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.082239 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.082175    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsl7\" (UniqueName: \"kubernetes.io/projected/81870999-5195-4f20-807d-c4adaf5c08bc-kube-api-access-hfsl7\") pod \"opendatahub-operator-controller-manager-77b67b5d96-z9kxp\" (UID: \"81870999-5195-4f20-807d-c4adaf5c08bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.134221 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.134189    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:27.626684 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:27.626596    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"]
Oct 30 21:53:27.629298 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:53:27.629275    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81870999_5195_4f20_807d_c4adaf5c08bc.slice/crio-beaaea35bf4f85e9e58d1cbb3102f4060226a9fb7d1a028edcc798393f1af6ce WatchSource:0}: Error finding container beaaea35bf4f85e9e58d1cbb3102f4060226a9fb7d1a028edcc798393f1af6ce: Status 404 returned error can't find the container with id beaaea35bf4f85e9e58d1cbb3102f4060226a9fb7d1a028edcc798393f1af6ce
Oct 30 21:53:28.313791 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:28.313755    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp" event={"ID":"81870999-5195-4f20-807d-c4adaf5c08bc","Type":"ContainerStarted","Data":"beaaea35bf4f85e9e58d1cbb3102f4060226a9fb7d1a028edcc798393f1af6ce"}
Oct 30 21:53:31.326086 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:31.326048    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp" event={"ID":"81870999-5195-4f20-807d-c4adaf5c08bc","Type":"ContainerStarted","Data":"e0fa2c45762297db0e1ae2e00c68cef2286cfbc379018aee60057056aedbc0c2"}
Oct 30 21:53:32.331349 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:32.331314    4295 generic.go:356] "Generic (PLEG): container finished" podID="81870999-5195-4f20-807d-c4adaf5c08bc" containerID="e0fa2c45762297db0e1ae2e00c68cef2286cfbc379018aee60057056aedbc0c2" exitCode=0
Oct 30 21:53:32.331752 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:32.331374    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp" event={"ID":"81870999-5195-4f20-807d-c4adaf5c08bc","Type":"ContainerDied","Data":"e0fa2c45762297db0e1ae2e00c68cef2286cfbc379018aee60057056aedbc0c2"}
Oct 30 21:53:33.337168 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:33.337133    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp" event={"ID":"81870999-5195-4f20-807d-c4adaf5c08bc","Type":"ContainerStarted","Data":"275af7834a358022049260936ccff35cb38612d2909261a3333fddc960f306aa"}
Oct 30 21:53:33.337593 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:33.337270    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:33.485954 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:33.485902    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp" podStartSLOduration=2.648212249 podStartE2EDuration="7.48588936s" totalImagesPullingTime="4.367537035s" totalInitContainerRuntime="470.140076ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:33.485061782 +0000 UTC m=+216.127655596" watchObservedRunningTime="2026-10-30 21:53:33.48588936 +0000 UTC m=+216.128483149"
Oct 30 21:53:44.343795 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:53:44.343714    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-z9kxp"
Oct 30 21:53:53.606523 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:53:53.606485    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:54:05.889040 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:05.888997    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"]
Oct 30 21:54:05.894844 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:05.894820    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:05.968809 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:05.968772    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"]
Oct 30 21:54:06.012725 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.012688    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tmp\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.012929 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.012740    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tls-certs\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.012929 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.012765    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lw5\" (UniqueName: \"kubernetes.io/projected/cc6c1b07-5564-47bd-9009-4ba1d388dec5-kube-api-access-78lw5\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.030642 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.030614    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:54:06.077517 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.077472    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s4xz8\""
Oct 30 21:54:06.077517 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.077505    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:54:06.113690 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.113648    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tmp\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.113863 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.113716    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tls-certs\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.113863 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.113741    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lw5\" (UniqueName: \"kubernetes.io/projected/cc6c1b07-5564-47bd-9009-4ba1d388dec5-kube-api-access-78lw5\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.116077 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.116050    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tmp\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.116389 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.116370    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6c1b07-5564-47bd-9009-4ba1d388dec5-tls-certs\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.178247 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.178222    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lw5\" (UniqueName: \"kubernetes.io/projected/cc6c1b07-5564-47bd-9009-4ba1d388dec5-kube-api-access-78lw5\") pod \"kube-auth-proxy-b76569bf4-j5z6p\" (UID: \"cc6c1b07-5564-47bd-9009-4ba1d388dec5\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.208484 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.208455    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"
Oct 30 21:54:06.590716 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:06.590693    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p"]
Oct 30 21:54:06.593355 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:54:06.593322    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c1b07_5564_47bd_9009_4ba1d388dec5.slice/crio-e06074b3cc5110c3089d002bf1341de0b635b42a70a00bcb580a0f1259c984c8 WatchSource:0}: Error finding container e06074b3cc5110c3089d002bf1341de0b635b42a70a00bcb580a0f1259c984c8: Status 404 returned error can't find the container with id e06074b3cc5110c3089d002bf1341de0b635b42a70a00bcb580a0f1259c984c8
Oct 30 21:54:07.454635 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:07.454598    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p" event={"ID":"cc6c1b07-5564-47bd-9009-4ba1d388dec5","Type":"ContainerStarted","Data":"e06074b3cc5110c3089d002bf1341de0b635b42a70a00bcb580a0f1259c984c8"}
Oct 30 21:54:09.463220 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:09.463186    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p" event={"ID":"cc6c1b07-5564-47bd-9009-4ba1d388dec5","Type":"ContainerStarted","Data":"57181bf472b1699ad4360d3b679187abb4a7f5b4b3141f58716f80ba42979640"}
Oct 30 21:54:09.583051 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:09.582952    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p" podStartSLOduration=1.8663862610000002 podStartE2EDuration="4.582936069s" totalImagesPullingTime="2.716549808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.582230792 +0000 UTC m=+252.224824595" watchObservedRunningTime="2026-10-30 21:54:09.582936069 +0000 UTC m=+252.225529895"
Oct 30 21:54:23.607560 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:54:23.607522    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:54:32.049997 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.049963    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"]
Oct 30 21:54:32.051857 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.051842    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.106357 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.106332    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 21:54:32.152320 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.152294    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:32.155467 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.155441    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:32.155576 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.155487    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n4kmw\""
Oct 30 21:54:32.156453 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.156435    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"]
Oct 30 21:54:32.212294 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.212263    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a903a436-87b2-4e71-b7b8-0f21affb8e70-data\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.212294 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.212295    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvj2s\" (UniqueName: \"kubernetes.io/projected/a903a436-87b2-4e71-b7b8-0f21affb8e70-kube-api-access-hvj2s\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.313554 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.313484    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a903a436-87b2-4e71-b7b8-0f21affb8e70-data\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.313554 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.313516    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvj2s\" (UniqueName: \"kubernetes.io/projected/a903a436-87b2-4e71-b7b8-0f21affb8e70-kube-api-access-hvj2s\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.313912 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.313893    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a903a436-87b2-4e71-b7b8-0f21affb8e70-data\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.377754 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.377729    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvj2s\" (UniqueName: \"kubernetes.io/projected/a903a436-87b2-4e71-b7b8-0f21affb8e70-kube-api-access-hvj2s\") pod \"postgres-7cdbf6dc44-54lpm\" (UID: \"a903a436-87b2-4e71-b7b8-0f21affb8e70\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:32.662498 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:32.662415    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:33.031482 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:33.031460    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"]
Oct 30 21:54:33.033944 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:54:33.033908    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda903a436_87b2_4e71_b7b8_0f21affb8e70.slice/crio-be861b34ca2aad88c5844ee6ddac00b07d327681ded77b4b79dae321ec9843d5 WatchSource:0}: Error finding container be861b34ca2aad88c5844ee6ddac00b07d327681ded77b4b79dae321ec9843d5: Status 404 returned error can't find the container with id be861b34ca2aad88c5844ee6ddac00b07d327681ded77b4b79dae321ec9843d5
Oct 30 21:54:33.546270 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:33.546234    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm" event={"ID":"a903a436-87b2-4e71-b7b8-0f21affb8e70","Type":"ContainerStarted","Data":"be861b34ca2aad88c5844ee6ddac00b07d327681ded77b4b79dae321ec9843d5"}
Oct 30 21:54:39.569761 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:39.569724    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm" event={"ID":"a903a436-87b2-4e71-b7b8-0f21affb8e70","Type":"ContainerStarted","Data":"d3a49b5286d0f856b7563a5e78e3895939e1d94a4088266433ff2df26e6f6c4d"}
Oct 30 21:54:39.570252 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:39.569859    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:39.688515 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:39.688463    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm" podStartSLOduration=1.544962075 podStartE2EDuration="7.688446809s" totalImagesPullingTime="6.143484734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:39.687472097 +0000 UTC m=+282.330065886" watchObservedRunningTime="2026-10-30 21:54:39.688446809 +0000 UTC m=+282.331040598"
Oct 30 21:54:45.601849 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:45.601817    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-54lpm"
Oct 30 21:54:53.606744 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:54:53.606706    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:54:54.715528 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.715495    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:54:54.719558 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.719540    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.778752 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.778723    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:54:54.784032 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.784004    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:54:54.784161 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.784146    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:54:54.823986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.823956    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-24lvx\""
Oct 30 21:54:54.823986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.823970    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:54:54.823986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.823973    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:54:54.825062 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.825044    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:54:54.885572 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.885535    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.885841 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.885649    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.885899 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.885877    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszbx\" (UniqueName: \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.987127 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.987060    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszbx\" (UniqueName: \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.987127 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.987109    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.987300 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.987138    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.989707 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.989682    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:54.989801 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:54.989710    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:55.050046 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.050017    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszbx\" (UniqueName: \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\") pod \"maas-controller-579cf7c6d-6vlcr\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") " pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:55.142560 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.142535    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 21:54:55.146387 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.146357    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.206838 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.206805    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 21:54:55.289267 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.289191    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.289267 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.289238    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.289449 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.289305    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz6v\" (UniqueName: \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.331187 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.331164    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:55.390163 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.390126    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.390287 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.390177    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.390287 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.390203    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz6v\" (UniqueName: \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.394340 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.394238    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.394450 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.394372    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.453871 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.453846    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz6v\" (UniqueName: \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\") pod \"maas-controller-5fb4f557f9-d8xc9\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") " pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.457706 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.457689    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:55.701978 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.701954    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:54:55.704722 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:54:55.704697    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9326feb5_9ba8_412d_b1ec_1fadda89b15d.slice/crio-811a5e56fac571df9f936744202aeb3ffd0eb1c8d0a94b33269c0891f53c3ebd WatchSource:0}: Error finding container 811a5e56fac571df9f936744202aeb3ffd0eb1c8d0a94b33269c0891f53c3ebd: Status 404 returned error can't find the container with id 811a5e56fac571df9f936744202aeb3ffd0eb1c8d0a94b33269c0891f53c3ebd
Oct 30 21:54:55.829840 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:55.829808    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 21:54:55.832043 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:54:55.832018    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd01aef_13c8_4346_a2cd_d3207ea4d7f2.slice/crio-24e9d37ad062f496981c0428bf4513d976ff88d91bc7d760bf256a499b5e2d64 WatchSource:0}: Error finding container 24e9d37ad062f496981c0428bf4513d976ff88d91bc7d760bf256a499b5e2d64: Status 404 returned error can't find the container with id 24e9d37ad062f496981c0428bf4513d976ff88d91bc7d760bf256a499b5e2d64
Oct 30 21:54:56.626455 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:56.626418    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" event={"ID":"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2","Type":"ContainerStarted","Data":"24e9d37ad062f496981c0428bf4513d976ff88d91bc7d760bf256a499b5e2d64"}
Oct 30 21:54:56.627419 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:56.627396    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" event={"ID":"9326feb5-9ba8-412d-b1ec-1fadda89b15d","Type":"ContainerStarted","Data":"811a5e56fac571df9f936744202aeb3ffd0eb1c8d0a94b33269c0891f53c3ebd"}
Oct 30 21:54:59.062338 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.062293    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:54:59.640677 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.640635    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" event={"ID":"9326feb5-9ba8-412d-b1ec-1fadda89b15d","Type":"ContainerStarted","Data":"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"}
Oct 30 21:54:59.640892 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.640770    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:54:59.642170 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.642143    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" event={"ID":"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2","Type":"ContainerStarted","Data":"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"}
Oct 30 21:54:59.642295 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.642207    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:54:59.762963 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.762909    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" podStartSLOduration=2.459288284 podStartE2EDuration="5.762895807s" totalImagesPullingTime="3.303607523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:59.762196943 +0000 UTC m=+302.404790731" watchObservedRunningTime="2026-10-30 21:54:59.762895807 +0000 UTC m=+302.405489596"
Oct 30 21:54:59.883871 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:54:59.883818    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" podStartSLOduration=1.604826052 podStartE2EDuration="4.883802787s" totalImagesPullingTime="3.278976735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:59.883246441 +0000 UTC m=+302.525840247" watchObservedRunningTime="2026-10-30 21:54:59.883802787 +0000 UTC m=+302.526396575"
Oct 30 21:55:02.582924 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.582884    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"]
Oct 30 21:55:02.586322 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.586281    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.640797 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.640774    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:55:02.655386 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.655361    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.655504 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.655436    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.655548 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.655532    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.655587 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.655566    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626sv\" (UniqueName: \"kubernetes.io/projected/708793d9-9768-4314-8207-383e31bfb542-kube-api-access-626sv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.690110 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.690081    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:55:02.690259 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.690163    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-hj6kn\""
Oct 30 21:55:02.690259 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.690183    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:55:02.691314 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.691297    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"]
Oct 30 21:55:02.756324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.756298    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.756486 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.756383    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.756486 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.756411    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.756486 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.756436    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626sv\" (UniqueName: \"kubernetes.io/projected/708793d9-9768-4314-8207-383e31bfb542-kube-api-access-626sv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.756486 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:02.756411    4295 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 21:55:02.756700 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:02.756528    4295 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 21:55:02.756700 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:02.756538    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source podName:708793d9-9768-4314-8207-383e31bfb542 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:03.256514 +0000 UTC m=+305.899107788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-5ncvs" (UID: "708793d9-9768-4314-8207-383e31bfb542") : configmap "usage-tenancy-proxy-source" not found
Oct 30 21:55:02.756700 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:02.756606    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert podName:708793d9-9768-4314-8207-383e31bfb542 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:03.2565869 +0000 UTC m=+305.899180673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-5ncvs" (UID: "708793d9-9768-4314-8207-383e31bfb542") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 21:55:02.756960 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.756942    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:02.821197 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:02.821165    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626sv\" (UniqueName: \"kubernetes.io/projected/708793d9-9768-4314-8207-383e31bfb542-kube-api-access-626sv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.260804 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.260766    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.260987 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.260821    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.261562 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.261545    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/708793d9-9768-4314-8207-383e31bfb542-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.263691 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.263657    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/708793d9-9768-4314-8207-383e31bfb542-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5ncvs\" (UID: \"708793d9-9768-4314-8207-383e31bfb542\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.496846 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.496817    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:03.867357 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.867293    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"]
Oct 30 21:55:03.869792 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:55:03.869766    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708793d9_9768_4314_8207_383e31bfb542.slice/crio-2e61bbd1e12bf68bd6be6f8f3f8ea5eecb9346c7dfa1040964ceaa4ecbd7f70a WatchSource:0}: Error finding container 2e61bbd1e12bf68bd6be6f8f3f8ea5eecb9346c7dfa1040964ceaa4ecbd7f70a: Status 404 returned error can't find the container with id 2e61bbd1e12bf68bd6be6f8f3f8ea5eecb9346c7dfa1040964ceaa4ecbd7f70a
Oct 30 21:55:03.871692 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:03.871656    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:55:04.661150 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:04.661113    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs" event={"ID":"708793d9-9768-4314-8207-383e31bfb542","Type":"ContainerStarted","Data":"2e61bbd1e12bf68bd6be6f8f3f8ea5eecb9346c7dfa1040964ceaa4ecbd7f70a"}
Oct 30 21:55:10.650585 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:10.649587    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 21:55:10.651389 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:10.651366    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:55:10.824605 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:10.824572    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:55:10.824881 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:10.824831    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" podUID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" containerName="manager" containerID="cri-o://980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400" gracePeriod=30
Oct 30 21:55:11.100490 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.100234    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:55:11.235105 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.235018    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\") pod \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") "
Oct 30 21:55:11.235265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.235129    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\" (UniqueName: \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\") pod \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") "
Oct 30 21:55:11.235265 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.235182    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\") pod \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\" (UID: \"9326feb5-9ba8-412d-b1ec-1fadda89b15d\") "
Oct 30 21:55:11.238490 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.238442    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs" pod "9326feb5-9ba8-412d-b1ec-1fadda89b15d" (UID: "9326feb5-9ba8-412d-b1ec-1fadda89b15d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:11.240246 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.240215    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx" pod "9326feb5-9ba8-412d-b1ec-1fadda89b15d" (UID: "9326feb5-9ba8-412d-b1ec-1fadda89b15d"). InnerVolumeSpecName "kube-api-access-kszbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:11.240809 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.240762    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert" pod "9326feb5-9ba8-412d-b1ec-1fadda89b15d" (UID: "9326feb5-9ba8-412d-b1ec-1fadda89b15d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:11.335898 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.335861    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:11.335898 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.335895    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kszbx\" (UniqueName: \"kubernetes.io/projected/9326feb5-9ba8-412d-b1ec-1fadda89b15d-kube-api-access-kszbx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:11.335898 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.335907    4295 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9326feb5-9ba8-412d-b1ec-1fadda89b15d-webhook-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:11.694505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.694232    4295 generic.go:356] "Generic (PLEG): container finished" podID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" containerID="980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400" exitCode=0
Oct 30 21:55:11.694505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.694288    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" event={"ID":"9326feb5-9ba8-412d-b1ec-1fadda89b15d","Type":"ContainerDied","Data":"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"}
Oct 30 21:55:11.694505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.694295    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-6vlcr"
Oct 30 21:55:11.694505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.694344    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-6vlcr" event={"ID":"9326feb5-9ba8-412d-b1ec-1fadda89b15d","Type":"ContainerDied","Data":"811a5e56fac571df9f936744202aeb3ffd0eb1c8d0a94b33269c0891f53c3ebd"}
Oct 30 21:55:11.694505 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.694375    4295 scope.go:122] "RemoveContainer" containerID="980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"
Oct 30 21:55:11.705012 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.704987    4295 scope.go:122] "RemoveContainer" containerID="980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"
Oct 30 21:55:11.705372 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:11.705335    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400\": container with ID starting with 980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400 not found: ID does not exist" containerID="980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"
Oct 30 21:55:11.705457 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.705385    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400"} err="failed to get container status \"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400\": rpc error: code = NotFound desc = could not find container \"980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400\": container with ID starting with 980f11174bfedf976d183fd3fc03be1b77a8154a61931b14527637c76bd8f400 not found: ID does not exist"
Oct 30 21:55:11.868538 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.868448    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:55:11.873976 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:11.873918    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-6vlcr"]
Oct 30 21:55:13.491397 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:13.491359    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" path="/var/lib/kubelet/pods/9326feb5-9ba8-412d-b1ec-1fadda89b15d/volumes"
Oct 30 21:55:17.723499 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:17.723465    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs" event={"ID":"708793d9-9768-4314-8207-383e31bfb542","Type":"ContainerStarted","Data":"0df5124055cee1b9324a4df170f22c60850e3a9a929d05292c9cbc77eae35ec9"}
Oct 30 21:55:17.723913 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:17.723597    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:17.845652 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:17.845600    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs" podStartSLOduration=2.678557577 podStartE2EDuration="15.845587245s" totalImagesPullingTime="13.167029668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:17.844954946 +0000 UTC m=+320.487548738" watchObservedRunningTime="2026-10-30 21:55:17.845587245 +0000 UTC m=+320.488181034"
Oct 30 21:55:23.607349 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:23.607313    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:55:28.732951 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:28.732920    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5ncvs"
Oct 30 21:55:36.136506 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.136467    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:36.136909 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.136794    4295 cpu_manager.go:414] "containerMap: removing container" podUID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" containerName="manager"
Oct 30 21:55:36.136909 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.136807    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" containerName="manager"
Oct 30 21:55:36.136909 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.136864    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="9326feb5-9ba8-412d-b1ec-1fadda89b15d" containerName="manager"
Oct 30 21:55:36.140905 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.140890    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.242839 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.242804    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.242999 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.242854    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq5x\" (UniqueName: \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.242999 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.242948    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.242999 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.242981    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.245284 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.245261    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:55:36.245366 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.245302    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:55:36.245366 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.245304    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\""
Oct 30 21:55:36.247141 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.247124    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:36.344029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.344005    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq5x\" (UniqueName: \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.344189 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.344051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.344189 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.344078    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.344189 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.344132    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.344746 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.344725    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.346707 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.346690    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.346780 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.346724    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.411136 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.411072    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq5x\" (UniqueName: \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\") pod \"maas-api-6d669c7fcd-wp4tf\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.451856 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.451825    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:36.825176 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:36.825059    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:36.827492 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:55:36.827467    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d336acf_fb62_403a_9f49_da00cb100f2c.slice/crio-1e22db20ea3f631d67e2ec40a65cf76c8ad895b7ef1a10faafee454d0775a0e9 WatchSource:0}: Error finding container 1e22db20ea3f631d67e2ec40a65cf76c8ad895b7ef1a10faafee454d0775a0e9: Status 404 returned error can't find the container with id 1e22db20ea3f631d67e2ec40a65cf76c8ad895b7ef1a10faafee454d0775a0e9
Oct 30 21:55:37.470036 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.469997    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"]
Oct 30 21:55:37.472898 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.472876    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.578617 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.578584    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"]
Oct 30 21:55:37.654998 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.654964    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655172 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655103    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655172 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655122    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655172 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655141    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef232b8-14d6-4220-aaba-ba04a64721ea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655268 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655244    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655304 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655284    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdlm\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-kube-api-access-ngdlm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.655336 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.655315    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.755888 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755804    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.755888 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755850    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.755888 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755880    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef232b8-14d6-4220-aaba-ba04a64721ea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.756165 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755909    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.756165 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755934    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdlm\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-kube-api-access-ngdlm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.756165 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.755964    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.756165 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.756029    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.757098 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.757069    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.758508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.758483    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef232b8-14d6-4220-aaba-ba04a64721ea-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.758863 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.758836    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.759163 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.759140    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.759263 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.759244    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.804716 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.804655    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" event={"ID":"0d336acf-fb62-403a-9f49-da00cb100f2c","Type":"ContainerStarted","Data":"1e22db20ea3f631d67e2ec40a65cf76c8ad895b7ef1a10faafee454d0775a0e9"}
Oct 30 21:55:37.821009 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.820981    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:37.821149 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:37.821131    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdlm\" (UniqueName: \"kubernetes.io/projected/aef232b8-14d6-4220-aaba-ba04a64721ea-kube-api-access-ngdlm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-72njd\" (UID: \"aef232b8-14d6-4220-aaba-ba04a64721ea\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:38.082474 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.082389    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:38.516351 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.516307    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"]
Oct 30 21:55:38.523548 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:55:38.523515    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef232b8_14d6_4220_aaba_ba04a64721ea.slice/crio-7fd1899b5bc0b67e0971673db0b825b5d0608a04e5f286d4e019dc4d04f5707e WatchSource:0}: Error finding container 7fd1899b5bc0b67e0971673db0b825b5d0608a04e5f286d4e019dc4d04f5707e: Status 404 returned error can't find the container with id 7fd1899b5bc0b67e0971673db0b825b5d0608a04e5f286d4e019dc4d04f5707e
Oct 30 21:55:38.526356 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.526318    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:55:38.526434 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.526406    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:55:38.810179 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.810140    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" event={"ID":"0d336acf-fb62-403a-9f49-da00cb100f2c","Type":"ContainerStarted","Data":"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"}
Oct 30 21:55:38.810324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.810247    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:38.811987 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.811961    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd" event={"ID":"aef232b8-14d6-4220-aaba-ba04a64721ea","Type":"ContainerStarted","Data":"52bc24db1e202460067b4a175e8f61e6469247cfb264ed1231a33f6354132c50"}
Oct 30 21:55:38.812082 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.812001    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd" event={"ID":"aef232b8-14d6-4220-aaba-ba04a64721ea","Type":"ContainerStarted","Data":"7fd1899b5bc0b67e0971673db0b825b5d0608a04e5f286d4e019dc4d04f5707e"}
Oct 30 21:55:38.812082 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.812020    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:38.931797 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:38.931747    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" podStartSLOduration=1.035187829 podStartE2EDuration="2.931732448s" totalImagesPullingTime="1.896544619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:38.930803749 +0000 UTC m=+341.573397537" watchObservedRunningTime="2026-10-30 21:55:38.931732448 +0000 UTC m=+341.574326263"
Oct 30 21:55:39.062353 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:39.062267    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd" podStartSLOduration=2.062253658 podStartE2EDuration="2.062253658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:39.061416475 +0000 UTC m=+341.704010274" watchObservedRunningTime="2026-10-30 21:55:39.062253658 +0000 UTC m=+341.704847447"
Oct 30 21:55:39.818329 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:39.818295    4295 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-72njd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:55:39.818772 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:39.818356    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd" podUID="aef232b8-14d6-4220-aaba-ba04a64721ea" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:55:42.817761 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:42.817730    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd"
Oct 30 21:55:43.058243 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.058213    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:55:43.058414 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.058394    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery" containerID="cri-o://a058dd19d68c8bb6d0932fcb67901c50c1b5aa1f885f8066e5a7bfd3915c18a9" gracePeriod=30
Oct 30 21:55:43.195581 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.195550    4295 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6qbp2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused" start-of-body=
Oct 30 21:55:43.195744 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.195608    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused"
Oct 30 21:55:43.834782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.834743    4295 generic.go:356] "Generic (PLEG): container finished" podID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerID="a058dd19d68c8bb6d0932fcb67901c50c1b5aa1f885f8066e5a7bfd3915c18a9" exitCode=0
Oct 30 21:55:43.834782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.834776    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" event={"ID":"e1f09cce-c09e-494c-930c-08fc6d0535af","Type":"ContainerDied","Data":"a058dd19d68c8bb6d0932fcb67901c50c1b5aa1f885f8066e5a7bfd3915c18a9"}
Oct 30 21:55:43.969485 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:43.969452    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:55:44.019446 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019412    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019601 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019467    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019601 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019510    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019601 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019583    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019775 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019620    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019775 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019692    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.019775 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.019741    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\") pod \"e1f09cce-c09e-494c-930c-08fc6d0535af\" (UID: \"e1f09cce-c09e-494c-930c-08fc6d0535af\") "
Oct 30 21:55:44.020348 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.020300    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:44.022057 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022030    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:44.022343 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022322    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "kube-api-access-sbxjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:44.022343 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022325    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:44.022446 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022371    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:44.022503 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022459    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:44.022536 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.022514    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig" pod "e1f09cce-c09e-494c-930c-08fc6d0535af" (UID: "e1f09cce-c09e-494c-930c-08fc6d0535af"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120703    4295 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-kubeconfig\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120734    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbxjt\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-kube-api-access-sbxjt\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120750    4295 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-ca-configmap\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120765    4295 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-cacerts\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120777    4295 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f09cce-c09e-494c-930c-08fc6d0535af-local-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.120786 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120787    4295 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-csr-dns-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.121153 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.120796    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f09cce-c09e-494c-930c-08fc6d0535af-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.821965 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.821937    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:44.840015 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.839992    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"
Oct 30 21:55:44.840015 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.840016    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2" event={"ID":"e1f09cce-c09e-494c-930c-08fc6d0535af","Type":"ContainerDied","Data":"1240c09bd11384b17f6d347afa9c5305d7f825f4209df6adc184b3561c95396f"}
Oct 30 21:55:44.840437 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:44.840069    4295 scope.go:122] "RemoveContainer" containerID="a058dd19d68c8bb6d0932fcb67901c50c1b5aa1f885f8066e5a7bfd3915c18a9"
Oct 30 21:55:45.059986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:45.059944    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:55:45.073810 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:45.073725    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2"]
Oct 30 21:55:45.491134 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:45.491101    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" path="/var/lib/kubelet/pods/e1f09cce-c09e-494c-930c-08fc6d0535af/volumes"
Oct 30 21:55:53.607279 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:53.607243    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:55:55.301156 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.301125    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"]
Oct 30 21:55:55.301529 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.301430    4295 cpu_manager.go:414] "containerMap: removing container" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery"
Oct 30 21:55:55.301529 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.301441    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery"
Oct 30 21:55:55.301529 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.301493    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f09cce-c09e-494c-930c-08fc6d0535af" containerName="discovery"
Oct 30 21:55:55.315950 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.315932    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.364489 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.364461    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"]
Oct 30 21:55:55.398182 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.398148    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/476f745d-edf3-453d-898c-02c0a9027515-manager-config\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.398322 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.398187    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-metrics-certs\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.398322 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.398220    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-cert\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.398442 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.398357    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwfk\" (UniqueName: \"kubernetes.io/projected/476f745d-edf3-453d-898c-02c0a9027515-kube-api-access-8dwfk\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.414012 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.413989    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:55:55.420293 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.420271    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4rxj7\""
Oct 30 21:55:55.420366 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.420340    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:55:55.420469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.420378    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:55:55.420469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.420397    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:55:55.420469 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.420416    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:55:55.499634 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.499609    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwfk\" (UniqueName: \"kubernetes.io/projected/476f745d-edf3-453d-898c-02c0a9027515-kube-api-access-8dwfk\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.499806 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.499649    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/476f745d-edf3-453d-898c-02c0a9027515-manager-config\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.499806 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.499693    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-metrics-certs\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.499806 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.499712    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-cert\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.500433 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.500412    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/476f745d-edf3-453d-898c-02c0a9027515-manager-config\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.502341 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.502323    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-metrics-certs\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.502485 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.502464    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/476f745d-edf3-453d-898c-02c0a9027515-cert\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.563890 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.563827    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwfk\" (UniqueName: \"kubernetes.io/projected/476f745d-edf3-453d-898c-02c0a9027515-kube-api-access-8dwfk\") pod \"jobset-controller-manager-769bdcb8b9-rj8wq\" (UID: \"476f745d-edf3-453d-898c-02c0a9027515\") " pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.626029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.626001    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:55:55.997170 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:55.996985    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"]
Oct 30 21:55:55.999738 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:55:55.999713    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476f745d_edf3_453d_898c_02c0a9027515.slice/crio-2a355e4e5361bb749ed0c0771fea3bd6225a5e4434638cb5db9975b4c556c508 WatchSource:0}: Error finding container 2a355e4e5361bb749ed0c0771fea3bd6225a5e4434638cb5db9975b4c556c508: Status 404 returned error can't find the container with id 2a355e4e5361bb749ed0c0771fea3bd6225a5e4434638cb5db9975b4c556c508
Oct 30 21:55:56.540384 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.540347    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:56.540853 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.540608    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" podUID="0d336acf-fb62-403a-9f49-da00cb100f2c" containerName="maas-api" containerID="cri-o://bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527" gracePeriod=30
Oct 30 21:55:56.773711 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.773689    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:56.810702 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.810625    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\" (UniqueName: \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\") pod \"0d336acf-fb62-403a-9f49-da00cb100f2c\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") "
Oct 30 21:55:56.810702 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.810684    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\") pod \"0d336acf-fb62-403a-9f49-da00cb100f2c\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") "
Oct 30 21:55:56.810875 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.810824    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\") pod \"0d336acf-fb62-403a-9f49-da00cb100f2c\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") "
Oct 30 21:55:56.810929 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.810903    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\") pod \"0d336acf-fb62-403a-9f49-da00cb100f2c\" (UID: \"0d336acf-fb62-403a-9f49-da00cb100f2c\") "
Oct 30 21:55:56.811188 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.811155    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca" pod "0d336acf-fb62-403a-9f49-da00cb100f2c" (UID: "0d336acf-fb62-403a-9f49-da00cb100f2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:56.812996 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.812975    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs" pod "0d336acf-fb62-403a-9f49-da00cb100f2c" (UID: "0d336acf-fb62-403a-9f49-da00cb100f2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:56.813068 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.813015    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x" pod "0d336acf-fb62-403a-9f49-da00cb100f2c" (UID: "0d336acf-fb62-403a-9f49-da00cb100f2c"). InnerVolumeSpecName "kube-api-access-rnq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:56.813112 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.813097    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls" pod "0d336acf-fb62-403a-9f49-da00cb100f2c" (UID: "0d336acf-fb62-403a-9f49-da00cb100f2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:56.887821 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.887792    4295 generic.go:356] "Generic (PLEG): container finished" podID="0d336acf-fb62-403a-9f49-da00cb100f2c" containerID="bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527" exitCode=0
Oct 30 21:55:56.887945 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.887838    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"
Oct 30 21:55:56.887945 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.887877    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" event={"ID":"0d336acf-fb62-403a-9f49-da00cb100f2c","Type":"ContainerDied","Data":"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"}
Oct 30 21:55:56.887945 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.887925    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf" event={"ID":"0d336acf-fb62-403a-9f49-da00cb100f2c","Type":"ContainerDied","Data":"1e22db20ea3f631d67e2ec40a65cf76c8ad895b7ef1a10faafee454d0775a0e9"}
Oct 30 21:55:56.888075 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.887948    4295 scope.go:122] "RemoveContainer" containerID="bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"
Oct 30 21:55:56.889268 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.889242    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq" event={"ID":"476f745d-edf3-453d-898c-02c0a9027515","Type":"ContainerStarted","Data":"2a355e4e5361bb749ed0c0771fea3bd6225a5e4434638cb5db9975b4c556c508"}
Oct 30 21:55:56.897311 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.897291    4295 scope.go:122] "RemoveContainer" containerID="bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"
Oct 30 21:55:56.897702 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:55:56.897615    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527\": container with ID starting with bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527 not found: ID does not exist" containerID="bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"
Oct 30 21:55:56.897702 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.897649    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527"} err="failed to get container status \"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527\": rpc error: code = NotFound desc = could not find container \"bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527\": container with ID starting with bb4b69b987a653197c8138c7efb1d512ed8ef6a2e2401bc945ac369dc3e3e527 not found: ID does not exist"
Oct 30 21:55:56.911454 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.911433    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d336acf-fb62-403a-9f49-da00cb100f2c-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.911454 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.911457    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.911578 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.911467    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnq5x\" (UniqueName: \"kubernetes.io/projected/0d336acf-fb62-403a-9f49-da00cb100f2c-kube-api-access-rnq5x\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.911578 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:56.911476    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d336acf-fb62-403a-9f49-da00cb100f2c-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:57.063708 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:57.063606    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:57.068872 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:57.068849    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wp4tf"]
Oct 30 21:55:57.491391 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:57.491353    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d336acf-fb62-403a-9f49-da00cb100f2c" path="/var/lib/kubelet/pods/0d336acf-fb62-403a-9f49-da00cb100f2c/volumes"
Oct 30 21:55:59.904084 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:59.904051    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq" event={"ID":"476f745d-edf3-453d-898c-02c0a9027515","Type":"ContainerStarted","Data":"4f1eb17f14e9c857f40df7576070f5bf16acd118b89754b4313c7e29fc18d1e1"}
Oct 30 21:55:59.904432 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:55:59.904198    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:56:00.029871 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:56:00.029822    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq" podStartSLOduration=1.225729787 podStartE2EDuration="5.029809289s" totalImagesPullingTime="3.804079502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:00.029116358 +0000 UTC m=+362.671710148" watchObservedRunningTime="2026-10-30 21:56:00.029809289 +0000 UTC m=+362.672403081"
Oct 30 21:56:10.914389 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:56:10.914357    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-769bdcb8b9-rj8wq"
Oct 30 21:56:23.607301 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:56:23.607259    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:56:53.607200 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:56:53.607160    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:57:23.607019 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:57:23.606980    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:57:53.607584 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:57:53.607544    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:58:02.489757 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.489721    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"]
Oct 30 21:58:02.490266 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.490008    4295 cpu_manager.go:414] "containerMap: removing container" podUID="0d336acf-fb62-403a-9f49-da00cb100f2c" containerName="maas-api"
Oct 30 21:58:02.490266 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.490018    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d336acf-fb62-403a-9f49-da00cb100f2c" containerName="maas-api"
Oct 30 21:58:02.490266 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.490083    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d336acf-fb62-403a-9f49-da00cb100f2c" containerName="maas-api"
Oct 30 21:58:02.492801 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.492785    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.596760 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.596725    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:58:02.596934 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.596763    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:58:02.596934 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.596867    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hwt6r\""
Oct 30 21:58:02.596934 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.596879    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:58:02.598608 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.598589    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"]
Oct 30 21:58:02.647708 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647657    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8e5465-9769-41b1-9526-75740bec3746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.647708 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647711    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.647948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647775    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.647948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647804    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.647948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647883    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ch7\" (UniqueName: \"kubernetes.io/projected/ac8e5465-9769-41b1-9526-75740bec3746-kube-api-access-k6ch7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.647948 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.647904    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748314 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748241    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ch7\" (UniqueName: \"kubernetes.io/projected/ac8e5465-9769-41b1-9526-75740bec3746-kube-api-access-k6ch7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748314 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748273    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748314 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748302    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8e5465-9769-41b1-9526-75740bec3746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748589 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748324    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748589 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748346    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748589 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748382    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748814 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748790    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748902 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748879    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.748954 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.748919    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.750876 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.750854    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8e5465-9769-41b1-9526-75740bec3746-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.751275 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.751255    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8e5465-9769-41b1-9526-75740bec3746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:02.816563 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:02.816536    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ch7\" (UniqueName: \"kubernetes.io/projected/ac8e5465-9769-41b1-9526-75740bec3746-kube-api-access-k6ch7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp\" (UID: \"ac8e5465-9769-41b1-9526-75740bec3746\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:03.105113 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:03.105034    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:03.477447 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:03.477393    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"]
Oct 30 21:58:03.480096 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:58:03.480074    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8e5465_9769_41b1_9526_75740bec3746.slice/crio-75914885d0ca4b2686db69e2f7289cb765502e4736493e35c1dc3c2453a56dea WatchSource:0}: Error finding container 75914885d0ca4b2686db69e2f7289cb765502e4736493e35c1dc3c2453a56dea: Status 404 returned error can't find the container with id 75914885d0ca4b2686db69e2f7289cb765502e4736493e35c1dc3c2453a56dea
Oct 30 21:58:04.380705 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:04.380646    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp" event={"ID":"ac8e5465-9769-41b1-9526-75740bec3746","Type":"ContainerStarted","Data":"75914885d0ca4b2686db69e2f7289cb765502e4736493e35c1dc3c2453a56dea"}
Oct 30 21:58:05.386369 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:05.386284    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp" event={"ID":"ac8e5465-9769-41b1-9526-75740bec3746","Type":"ContainerStarted","Data":"9d5f3b555bfe3091c6d1c382aea193ad8408411ddfd042289fe2f0365d03742e"}
Oct 30 21:58:05.508518 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:05.508465    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp" podStartSLOduration=1.936291717 podStartE2EDuration="3.508450478s" totalImagesPullingTime="1.572158761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:05.507723714 +0000 UTC m=+488.150317504" watchObservedRunningTime="2026-10-30 21:58:05.508450478 +0000 UTC m=+488.151044267"
Oct 30 21:58:06.105683 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:06.105615    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:07.483728 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.483695    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"]
Oct 30 21:58:07.488320 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.488303    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584344 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584311    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584344 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584345    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584553 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584489    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d373d5-5760-4d37-8351-8668586c8f88-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584553 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584527    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584553 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584551    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.584694 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.584652    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrvg\" (UniqueName: \"kubernetes.io/projected/69d373d5-5760-4d37-8351-8668586c8f88-kube-api-access-blrvg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.592081 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.592061    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 21:58:07.593343 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.593324    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"]
Oct 30 21:58:07.686021 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.685986    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d373d5-5760-4d37-8351-8668586c8f88-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686271 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686249    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686441 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686406    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686539 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686443    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrvg\" (UniqueName: \"kubernetes.io/projected/69d373d5-5760-4d37-8351-8668586c8f88-kube-api-access-blrvg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686539 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686514    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686539    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686770 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686703    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.686939 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686906    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.687029 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.686939    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.688755 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.688735    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d373d5-5760-4d37-8351-8668586c8f88-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.689227 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.689209    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d373d5-5760-4d37-8351-8668586c8f88-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.753798 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.753729    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrvg\" (UniqueName: \"kubernetes.io/projected/69d373d5-5760-4d37-8351-8668586c8f88-kube-api-access-blrvg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn\" (UID: \"69d373d5-5760-4d37-8351-8668586c8f88\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:07.810714 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:07.810675    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:08.187206 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:08.187036    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"]
Oct 30 21:58:08.189813 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:58:08.189785    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69d373d5_5760_4d37_8351_8668586c8f88.slice/crio-89d92fa80d276d788451d163d296ffad7277c770c929f1115fa7c78e52b383b2 WatchSource:0}: Error finding container 89d92fa80d276d788451d163d296ffad7277c770c929f1115fa7c78e52b383b2: Status 404 returned error can't find the container with id 89d92fa80d276d788451d163d296ffad7277c770c929f1115fa7c78e52b383b2
Oct 30 21:58:08.397884 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:08.397854    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" event={"ID":"69d373d5-5760-4d37-8351-8668586c8f88","Type":"ContainerStarted","Data":"89d92fa80d276d788451d163d296ffad7277c770c929f1115fa7c78e52b383b2"}
Oct 30 21:58:09.404238 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:09.404188    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" event={"ID":"69d373d5-5760-4d37-8351-8668586c8f88","Type":"ContainerStarted","Data":"5591b896961d191db5f9577b86f915e1ddad2ff75800414e4bc0832f48fd7a05"}
Oct 30 21:58:09.525371 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:09.525319    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podStartSLOduration=1.949857535 podStartE2EDuration="2.525305906s" totalImagesPullingTime="575.448371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:09.524376775 +0000 UTC m=+492.166970590" watchObservedRunningTime="2026-10-30 21:58:09.525305906 +0000 UTC m=+492.167899695"
Oct 30 21:58:09.811796 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:09.811760    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:13.106258 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:13.106219    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:13.118617 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:13.118595    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:13.432731 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:13.432708    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp"
Oct 30 21:58:14.546241 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.546211    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"]
Oct 30 21:58:14.549298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.549278    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.645968 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.645931    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vl4w\" (UniqueName: \"kubernetes.io/projected/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-kube-api-access-6vl4w\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.645968 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.645969    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.646176 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.645991    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.646176 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.646129    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.646176 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.646150    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.646332 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.646244    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.653142 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.653119    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 21:58:14.654531 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.654513    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"]
Oct 30 21:58:14.746848 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746820    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746876    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746900    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vl4w\" (UniqueName: \"kubernetes.io/projected/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-kube-api-access-6vl4w\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746918    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746935    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.746982    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747308 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.747282    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747370 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.747297    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.747413 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.747387    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.749408 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.749387    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.749585 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.749567    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.811414 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.811351    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vl4w\" (UniqueName: \"kubernetes.io/projected/dc9c6d09-76e0-4fd0-912e-b4f7c9817890-kube-api-access-6vl4w\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76862\" (UID: \"dc9c6d09-76e0-4fd0-912e-b4f7c9817890\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:14.860538 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:14.860503    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:15.240089 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:15.240057    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"]
Oct 30 21:58:15.242956 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:58:15.242926    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9c6d09_76e0_4fd0_912e_b4f7c9817890.slice/crio-46dbbe756d342f4e01c608fc19107161854440cc4f32b59c143065cd221426bb WatchSource:0}: Error finding container 46dbbe756d342f4e01c608fc19107161854440cc4f32b59c143065cd221426bb: Status 404 returned error can't find the container with id 46dbbe756d342f4e01c608fc19107161854440cc4f32b59c143065cd221426bb
Oct 30 21:58:15.430893 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:15.430861    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862" event={"ID":"dc9c6d09-76e0-4fd0-912e-b4f7c9817890","Type":"ContainerStarted","Data":"46dbbe756d342f4e01c608fc19107161854440cc4f32b59c143065cd221426bb"}
Oct 30 21:58:16.436017 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:16.435975    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862" event={"ID":"dc9c6d09-76e0-4fd0-912e-b4f7c9817890","Type":"ContainerStarted","Data":"36db77685f951e748752a8906ae7387ae5f8e1996918a32f593d54d93e2ac30c"}
Oct 30 21:58:16.557986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:16.557936    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862" podStartSLOduration=1.9765797059999999 podStartE2EDuration="2.557921284s" totalImagesPullingTime="581.341578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:16.556744992 +0000 UTC m=+499.199338781" watchObservedRunningTime="2026-10-30 21:58:16.557921284 +0000 UTC m=+499.200515073"
Oct 30 21:58:16.861051 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:16.860964    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:17.492852 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.492817    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 21:58:17.496051 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.496028    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.551204 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.551178    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gbj8s\""
Oct 30 21:58:17.602137 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.602111    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 21:58:17.672449 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672422    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672602 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672459    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672602 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672483    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672602 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672550    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2mz\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672602 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672594    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672866 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672620    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672866 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672764    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672866 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672795    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.672866 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.672827    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773740 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773649    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773740 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773707    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773740 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773731    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773757    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773823    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773859    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773891    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773916    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2mz\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.773991 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.773942    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.774407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.774129    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.774407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.774168    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.774407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.774386    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.774527 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.774461    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.774722 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.774704    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.776315 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.776295    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.776444 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.776421    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.811624 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.811596    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:17.824362 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.824342    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:17.837285 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.837263    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:17.837437 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:17.837423    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2mz\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\") pod \"maas-default-gateway-openshift-default-67bf986f99-dv8z5\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:18.108330 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:18.108246    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:18.454983 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:18.454958    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 21:58:18.630685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:18.630549    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 21:58:18.633118 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:58:18.633090    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1001cfb_75b3_49cb_89d1_d14d2333828d.slice/crio-8a5ab51a0ab3fe8a36c7af4e7e839118a2fbd1f96a15e1a865a6bd9242b1a959 WatchSource:0}: Error finding container 8a5ab51a0ab3fe8a36c7af4e7e839118a2fbd1f96a15e1a865a6bd9242b1a959: Status 404 returned error can't find the container with id 8a5ab51a0ab3fe8a36c7af4e7e839118a2fbd1f96a15e1a865a6bd9242b1a959
Oct 30 21:58:19.448867 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:19.448823    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" event={"ID":"a1001cfb-75b3-49cb-89d1-d14d2333828d","Type":"ContainerStarted","Data":"8a5ab51a0ab3fe8a36c7af4e7e839118a2fbd1f96a15e1a865a6bd9242b1a959"}
Oct 30 21:58:22.449822 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:22.449789    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:22.450178 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:22.449857    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:22.450178 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:22.449888    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:23.471200 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:23.471161    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" event={"ID":"a1001cfb-75b3-49cb-89d1-d14d2333828d","Type":"ContainerStarted","Data":"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"}
Oct 30 21:58:23.631956 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:58:23.631922    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:58:23.647424 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:23.647374    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podStartSLOduration=2.833813262 podStartE2EDuration="6.647361402s" totalImagesPullingTime="3.81354814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.646583995 +0000 UTC m=+506.289177785" watchObservedRunningTime="2026-10-30 21:58:23.647361402 +0000 UTC m=+506.289955280"
Oct 30 21:58:24.112825 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:24.112772    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:24.114465 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:24.114440    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.114578 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:24.114495    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:24.860734 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:24.860701    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:24.873006 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:24.872981    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:25.109685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:25.109049    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.109685 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:25.109134    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:25.515316 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:25.515290    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76862"
Oct 30 21:58:26.109101 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:26.109057    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.109722 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:26.109688    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:27.112690 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:27.109656    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.112690 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:27.109746    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:28.112022 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:28.111984    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.112302 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:28.112271    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:29.109481 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:29.109446    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.109901 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:29.109503    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 21:58:30.590274 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:30.590241    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:30.590913 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:30.590889    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:30.591868 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:30.591848    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 21:58:32.629898 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.629869    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"]
Oct 30 21:58:32.633281 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.633264    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691471 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691446    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691475    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691495    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafe4d11-269b-499c-a118-520bdf759084-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691599 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691580    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafe4d11-269b-499c-a118-520bdf759084-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691793 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691625    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691793 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691650    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691793 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691724    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjwf\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-kube-api-access-hgjwf\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691793 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691756    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.691793 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.691772    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.792759 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.792722    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.792759 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.792758    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.792989 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.792778    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjwf\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-kube-api-access-hgjwf\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.792989 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.792911    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.792989 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.792949    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793147 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793022    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793147 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793147 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793079    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafe4d11-269b-499c-a118-520bdf759084-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793147 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793128    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafe4d11-269b-499c-a118-520bdf759084-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793316 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793172    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793351 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793314    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793454 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793433    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.793501 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.793442    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.794056 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.794033    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafe4d11-269b-499c-a118-520bdf759084-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.795324 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.795305    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafe4d11-269b-499c-a118-520bdf759084-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.795520 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.795504    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafe4d11-269b-499c-a118-520bdf759084-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.839016 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.838987    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"]
Oct 30 21:58:32.859923 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.859896    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.860422 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.860395    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjwf\" (UniqueName: \"kubernetes.io/projected/dafe4d11-269b-499c-a118-520bdf759084-kube-api-access-hgjwf\") pod \"maas-default-gateway-openshift-default-67bf986f99-vmh57\" (UID: \"dafe4d11-269b-499c-a118-520bdf759084\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:32.945891 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:32.945858    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:33.371215 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.371141    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"]
Oct 30 21:58:33.407925 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:58:33.407896    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafe4d11_269b_499c_a118_520bdf759084.slice/crio-2ca65d5750f0ba768eef771ae989c0f89ad0e067358d7d9503b806a01bda52cb WatchSource:0}: Error finding container 2ca65d5750f0ba768eef771ae989c0f89ad0e067358d7d9503b806a01bda52cb: Status 404 returned error can't find the container with id 2ca65d5750f0ba768eef771ae989c0f89ad0e067358d7d9503b806a01bda52cb
Oct 30 21:58:33.409903 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.409869    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.409986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.409932    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.409986 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.409961    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.514165 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.514116    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" event={"ID":"dafe4d11-269b-499c-a118-520bdf759084","Type":"ContainerStarted","Data":"2e050a6630e44cbcbccdce281e707e1d16ce89a29e4450d4e97e4684d1671110"}
Oct 30 21:58:33.514351 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.514187    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" event={"ID":"dafe4d11-269b-499c-a118-520bdf759084","Type":"ContainerStarted","Data":"2ca65d5750f0ba768eef771ae989c0f89ad0e067358d7d9503b806a01bda52cb"}
Oct 30 21:58:33.642161 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.642065    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podStartSLOduration=1.642047145 podStartE2EDuration="1.642047145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:33.640657799 +0000 UTC m=+516.283251600" watchObservedRunningTime="2026-10-30 21:58:33.642047145 +0000 UTC m=+516.284641118"
Oct 30 21:58:33.946182 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:33.946153    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:34.947639 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:34.947604    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:34.948089 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:34.947671    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:35.946337 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:35.946301    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:35.946512 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:35.946355    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:36.946534 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:36.946495    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:36.946968 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:36.946551    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:37.946504 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:37.946463    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:37.946698 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:37.946538    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:38.946311 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:38.946275    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:38.946488 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:38.946356    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:39.946515 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:39.946476    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 21:58:39.947082 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:39.946527    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 21:58:41.612858 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:41.612824    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:41.613545 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:41.613518    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:41.614563 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:58:41.614539    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57"
Oct 30 21:58:53.608101 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:58:53.608065    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:59:23.605979 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:59:23.605944    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:59:42.212073 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.211998    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 21:59:42.215356 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.215341    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.292532 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292497    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.292782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292558    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.292782 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292619    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.292896 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292815    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.292896 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292870    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.293001 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292928    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.293001 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.292966    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfj6k\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.293096 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.293075    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.293140 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.293118    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.295137 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.295111    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 21:59:42.320229 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.320199    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d8837a-openshift-default-dockercfg-p6b8g\""
Oct 30 21:59:42.394330 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394302    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394346    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394371    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394387    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394420    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394445    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394464    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394479    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfj6k\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394510 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394499    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394963 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394791    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.394963 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.394850    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.395079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.395055    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.395181 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.395164    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.395472 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.395447    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.396914 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.396896    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.397024 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.397007    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.461045 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.461021    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.461941 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.461913    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfj6k\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.527725 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.527621    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:42.894882 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.894809    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 21:59:42.899025 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.899008    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:42.953765 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.953736    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8ca52a-openshift-default-dockercfg-fdgr2\""
Oct 30 21:59:42.974254 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:59:42.974220    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86b46f8_0826_42d7_ad00_ba0578ba6a2d.slice/crio-884956e2fc192f9db8a329f22d707c2b7bc37c58e6ffb85ef391e405f1c3f7cc WatchSource:0}: Error finding container 884956e2fc192f9db8a329f22d707c2b7bc37c58e6ffb85ef391e405f1c3f7cc: Status 404 returned error can't find the container with id 884956e2fc192f9db8a329f22d707c2b7bc37c58e6ffb85ef391e405f1c3f7cc
Oct 30 21:59:42.976508 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.976474    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.976597 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.976539    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.976597 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.976571    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.982289 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:42.982264    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 21:59:43.000066 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000042    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000172 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000076    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzw4\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000249 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000177    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000249 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000223    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000333 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000294    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000382 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000355    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000457 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000440    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000498 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000470    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.000537 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.000528    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.004582 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.004561    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 21:59:43.101416 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101387    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101416 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101419    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101709 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101448    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101709 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101471    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101709 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101489    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzw4\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101709 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101601    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101709 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101645    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101992 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101711    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.101992 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.101817    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.102100 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.102012    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.102153 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.102102    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.102256 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.102235    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.102498 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.102477    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.102595 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.102576    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.104254 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.104228    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.104364 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.104271    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.165004 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.164925    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.165135 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.165015    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzw4\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.211216 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.211183    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:43.569139 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.569107    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 21:59:43.573438 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.573418    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.627731 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.627706    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2182a5-openshift-default-dockercfg-ssq65\""
Oct 30 21:59:43.646719 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.646684    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 21:59:43.649092 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:59:43.649068    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bab16b_a784_4c94_8ad7_a9bd39597bf6.slice/crio-cf6aaf436c5030abd3b6bf6357d2e19ee5974c6fa8aab4c89ff503220c98b247 WatchSource:0}: Error finding container cf6aaf436c5030abd3b6bf6357d2e19ee5974c6fa8aab4c89ff503220c98b247: Status 404 returned error can't find the container with id cf6aaf436c5030abd3b6bf6357d2e19ee5974c6fa8aab4c89ff503220c98b247
Oct 30 21:59:43.651501 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.651460    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.651605 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.651582    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.651678 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.651643    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.678702 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.678678    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 21:59:43.710008 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.709986    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710098 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710015    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710098 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710034    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710168 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710146    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710205 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710187    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710235 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710213    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710267 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710253    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chtr\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710298 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710274    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.710375 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.710362    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.810749 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.806321    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerStarted","Data":"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"}
Oct 30 21:59:43.810749 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.806383    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerStarted","Data":"884956e2fc192f9db8a329f22d707c2b7bc37c58e6ffb85ef391e405f1c3f7cc"}
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.811362    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerStarted","Data":"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"}
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.811429    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerStarted","Data":"cf6aaf436c5030abd3b6bf6357d2e19ee5974c6fa8aab4c89ff503220c98b247"}
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812738    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812783    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812807    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812833    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812884    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812929    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.812959    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.813019    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chtr\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.813056    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.814427 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.813468    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.815160 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.814455    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.815160 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.814679    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.815274 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.815169    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.815778 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.815727    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.818076 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.818050    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.821252 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.821178    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.879079 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.879048    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.880253 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.880235    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chtr\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.886116 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.886096    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:43.942484 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:43.942422    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podStartSLOduration=1.942403853 podStartE2EDuration="1.942403853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:43.940292795 +0000 UTC m=+586.582886588" watchObservedRunningTime="2026-10-30 21:59:43.942403853 +0000 UTC m=+586.584997645"
Oct 30 21:59:44.066114 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.066048    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podStartSLOduration=2.066026899 podStartE2EDuration="2.066026899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:44.064568172 +0000 UTC m=+586.707161962" watchObservedRunningTime="2026-10-30 21:59:44.066026899 +0000 UTC m=+586.708620690"
Oct 30 21:59:44.214694 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.212019    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:44.218136 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.218095    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:44.282935 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.282912    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 21:59:44.284975 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:59:44.284950    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6736d70_17a0_4076_9119_6fd677801fe1.slice/crio-92f09e078c767b7ca8d6593b4555e7f1270fab189816879e17a5322df19b935e WatchSource:0}: Error finding container 92f09e078c767b7ca8d6593b4555e7f1270fab189816879e17a5322df19b935e: Status 404 returned error can't find the container with id 92f09e078c767b7ca8d6593b4555e7f1270fab189816879e17a5322df19b935e
Oct 30 21:59:44.287183 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.287150    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.287258 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.287213    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.287258 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.287242    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.527899 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.527806    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:44.533516 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.533493    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:44.816954 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.816855    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerStarted","Data":"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"}
Oct 30 21:59:44.817407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.817296    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:44.817407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.817337    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:44.817407 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.817361    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerStarted","Data":"92f09e078c767b7ca8d6593b4555e7f1270fab189816879e17a5322df19b935e"}
Oct 30 21:59:44.817964 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.817944    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 21:59:44.818076 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.818062    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 21:59:44.886202 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.886171    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:44.942765 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:44.942711    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podStartSLOduration=1.942697902 podStartE2EDuration="1.942697902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:44.942305324 +0000 UTC m=+587.584899137" watchObservedRunningTime="2026-10-30 21:59:44.942697902 +0000 UTC m=+587.585291690"
Oct 30 21:59:45.891273 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:45.891243    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:46.826034 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:46.826003    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:46.827000 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:46.826979    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 21:59:53.058081 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.058048    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 21:59:53.061830 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.061811    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.116318 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.116286    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-d8837a-serving-cert\""
Oct 30 21:59:53.166840 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.166816    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:59:53.166999 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.166915    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\""
Oct 30 21:59:53.168418 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.168399    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 21:59:53.222222 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.222193    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.222360 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.222236    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.222398 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.222364    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.222398 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.222382    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g82q\" (UniqueName: \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.323771 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.323700    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g82q\" (UniqueName: \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.323771 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.323751    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.323949 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.323786    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.323949 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.323886    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.323949 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:59:53.323926    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-serving-cert: secret "maas-api-e2e-worker-w2-d8837a-serving-cert" not found
Oct 30 21:59:53.324062 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:59:53.323995    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls podName:b3247850-8f20-4132-8fd7-3cdc84e454ad nodeName:}" failed. No retries permitted until 2026-10-30 21:59:53.823974767 +0000 UTC m=+596.466568539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls") pod "maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" (UID: "b3247850-8f20-4132-8fd7-3cdc84e454ad") : secret "maas-api-e2e-worker-w2-d8837a-serving-cert" not found
Oct 30 21:59:53.324939 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.324916    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.326591 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.326567    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.391597 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.391565    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g82q\" (UniqueName: \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.611402 ip-10-0-135-59 kubenswrapper[4295]: E1030 21:59:53.611284    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 21:59:53.829044 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.829012    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.831637 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.831610    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:53.974824 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:53.974791    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:54.349130 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:54.348970    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 21:59:54.352041 ip-10-0-135-59 kubenswrapper[4295]: W1030 21:59:54.352017    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3247850_8f20_4132_8fd7_3cdc84e454ad.slice/crio-159f7ccbb0dcd06b6560f6dd41ce73bc374ef9e81906bf204da4d0f2cd16e60c WatchSource:0}: Error finding container 159f7ccbb0dcd06b6560f6dd41ce73bc374ef9e81906bf204da4d0f2cd16e60c: Status 404 returned error can't find the container with id 159f7ccbb0dcd06b6560f6dd41ce73bc374ef9e81906bf204da4d0f2cd16e60c
Oct 30 21:59:54.860396 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:54.860357    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" event={"ID":"b3247850-8f20-4132-8fd7-3cdc84e454ad","Type":"ContainerStarted","Data":"159f7ccbb0dcd06b6560f6dd41ce73bc374ef9e81906bf204da4d0f2cd16e60c"}
Oct 30 21:59:55.866517 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:55.866484    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" event={"ID":"b3247850-8f20-4132-8fd7-3cdc84e454ad","Type":"ContainerStarted","Data":"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"}
Oct 30 21:59:55.866968 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:55.866626    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 21:59:56.005623 ip-10-0-135-59 kubenswrapper[4295]: I1030 21:59:56.005567    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" podStartSLOduration=2.207801465 podStartE2EDuration="3.005547928s" totalImagesPullingTime="797.746463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:56.004128684 +0000 UTC m=+598.646722489" watchObservedRunningTime="2026-10-30 21:59:56.005547928 +0000 UTC m=+598.648141729"
Oct 30 22:00:00.208538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.208495    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:00.212498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.212474    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.215350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.215326    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.215473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.215372    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqkj\" (UniqueName: \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.272094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.272061    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:00.272308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.272280    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:00:00.316687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.316629    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.316826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.316700    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqkj\" (UniqueName: \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.317219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.317201    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.379952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.379920    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqkj\" (UniqueName: \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.524965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.524885    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:00.900173 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:00.900151    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:00.901818 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:00.901791    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c1032d_5e0a_475f_9eb7_43f0402942c9.slice/crio-5b69276f6d943a8525550823c1bf5840f483988b292199c85f6c3a3cdf7c34f0 WatchSource:0}: Error finding container 5b69276f6d943a8525550823c1bf5840f483988b292199c85f6c3a3cdf7c34f0: Status 404 returned error can't find the container with id 5b69276f6d943a8525550823c1bf5840f483988b292199c85f6c3a3cdf7c34f0
Oct 30 22:00:01.877527 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:01.877499    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 22:00:01.895851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:01.895820    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerStarted","Data":"5b69276f6d943a8525550823c1bf5840f483988b292199c85f6c3a3cdf7c34f0"}
Oct 30 22:00:02.380839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.380805    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:00:02.385509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.385493    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.434015 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.433980    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c26z\" (UniqueName: \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.434173 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.434108    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.450594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.450568    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-2182a5\""
Oct 30 22:00:02.450736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.450722    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-2182a5-dockercfg-8qprs\""
Oct 30 22:00:02.474221 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.474178    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:00:02.534867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.534817    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.535031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.534926    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c26z\" (UniqueName: \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.539076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.539045    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.558922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.558885    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:00:02.562360 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.562344    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.608425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.608391    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c26z\" (UniqueName: \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\") pod \"payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.629148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.629119    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:00:02.636585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.636507    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.636742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.636686    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbprj\" (UniqueName: \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.698238 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.698196    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:02.737147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.737107    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbprj\" (UniqueName: \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.737147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.737159    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.738853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.738201    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.825688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.819436    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbprj\" (UniqueName: \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\") pod \"payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:02.874879 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:02.874846    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:03.086311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:03.086282    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:00:03.086962 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:03.086937    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637c3951_da90_46f4_b6a3_092b3d4cc3c4.slice/crio-bd1bc98e8f1f28302863592104f097a668b8d66444f273a57be639cd11212a20 WatchSource:0}: Error finding container bd1bc98e8f1f28302863592104f097a668b8d66444f273a57be639cd11212a20: Status 404 returned error can't find the container with id bd1bc98e8f1f28302863592104f097a668b8d66444f273a57be639cd11212a20
Oct 30 22:00:03.270096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:03.270025    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:00:03.272545 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:03.272519    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3562e_550c_4567_be9a_cdad66c4d393.slice/crio-ca6586c8b3fae33c0a31b5e1ebbbad770073e56ad44c2d413faf4ecc0fe26179 WatchSource:0}: Error finding container ca6586c8b3fae33c0a31b5e1ebbbad770073e56ad44c2d413faf4ecc0fe26179: Status 404 returned error can't find the container with id ca6586c8b3fae33c0a31b5e1ebbbad770073e56ad44c2d413faf4ecc0fe26179
Oct 30 22:00:03.941681 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:03.941416    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" event={"ID":"637c3951-da90-46f4-b6a3-092b3d4cc3c4","Type":"ContainerStarted","Data":"bd1bc98e8f1f28302863592104f097a668b8d66444f273a57be639cd11212a20"}
Oct 30 22:00:03.943430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:03.943393    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" event={"ID":"0bd3562e-550c-4567-be9a-cdad66c4d393","Type":"ContainerStarted","Data":"ca6586c8b3fae33c0a31b5e1ebbbad770073e56ad44c2d413faf4ecc0fe26179"}
Oct 30 22:00:05.413691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.411563    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:00:05.420863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.420836    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.487385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.487344    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lrb\" (UniqueName: \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.487573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.487411    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.579636 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.579600    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:00:05.614735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.613201    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lrb\" (UniqueName: \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.614735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.613277    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.623005 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.622975    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bce513\""
Oct 30 22:00:05.623154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.623084    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bce513-dockercfg-bvrlx\""
Oct 30 22:00:05.625119 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.625007    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.679068 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.679036    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lrb\" (UniqueName: \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\") pod \"payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:05.738072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:05.738035    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:06.162265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:06.162126    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:00:06.166644 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:06.166607    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d64be76_3b79_413a_869d_4d9021a06e87.slice/crio-f9aa55c09c7bf1332083d7b87faec97d887640086abbb1ba3a646ad95df909c0 WatchSource:0}: Error finding container f9aa55c09c7bf1332083d7b87faec97d887640086abbb1ba3a646ad95df909c0: Status 404 returned error can't find the container with id f9aa55c09c7bf1332083d7b87faec97d887640086abbb1ba3a646ad95df909c0
Oct 30 22:00:06.169091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:06.169076    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:00:06.970754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:06.970714    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" event={"ID":"1d64be76-3b79-413a-869d-4d9021a06e87","Type":"ContainerStarted","Data":"f9aa55c09c7bf1332083d7b87faec97d887640086abbb1ba3a646ad95df909c0"}
Oct 30 22:00:07.978310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.978272    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" event={"ID":"0bd3562e-550c-4567-be9a-cdad66c4d393","Type":"ContainerStarted","Data":"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"}
Oct 30 22:00:07.978796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.978552    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:07.980536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.980512    4295 generic.go:356] "Generic (PLEG): container finished" podID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerID="151feb27b808953e30c4a4f6c7d715e37a9e8be90d379a42f9e06c88bb2f208c" exitCode=60
Oct 30 22:00:07.980711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.980687    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerDied","Data":"151feb27b808953e30c4a4f6c7d715e37a9e8be90d379a42f9e06c88bb2f208c"}
Oct 30 22:00:07.981022 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.981004    4295 scope.go:122] "RemoveContainer" containerID="151feb27b808953e30c4a4f6c7d715e37a9e8be90d379a42f9e06c88bb2f208c"
Oct 30 22:00:07.984001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.983973    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" event={"ID":"637c3951-da90-46f4-b6a3-092b3d4cc3c4","Type":"ContainerStarted","Data":"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"}
Oct 30 22:00:07.984106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:07.984096    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:08.113717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.109497    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:00:08.119203 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.119170    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.158117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.158075    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.158292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.158129    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf57w\" (UniqueName: \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.173332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.173266    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" podStartSLOduration=1.6205025320000002 podStartE2EDuration="6.173248639s" totalImagesPullingTime="4.552746107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.133389173 +0000 UTC m=+610.775982962" watchObservedRunningTime="2026-10-30 22:00:08.173248639 +0000 UTC m=+610.815842429"
Oct 30 22:00:08.174072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.174048    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:00:08.181650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.181629    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-9cff60-dockercfg-pllnt\""
Oct 30 22:00:08.201943 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.200006    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9cff60\""
Oct 30 22:00:08.264092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.262199    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.264092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.262256    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf57w\" (UniqueName: \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.264092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.263868    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.279029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.278966    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" podStartSLOduration=1.591521857 podStartE2EDuration="6.278947399s" totalImagesPullingTime="4.687425542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.277335007 +0000 UTC m=+610.919928796" watchObservedRunningTime="2026-10-30 22:00:08.278947399 +0000 UTC m=+610.921541188"
Oct 30 22:00:08.327092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.327061    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf57w\" (UniqueName: \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\") pod \"payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.435703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.434539    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:08.841081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.841049    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:00:08.841893 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:08.841869    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bf5c9_2b25_407f_8a86_896031ab5184.slice/crio-fef79d49a6eedf266893ced850f2fb68f3555c2b1d224b9a356e36e1865bfaa7 WatchSource:0}: Error finding container fef79d49a6eedf266893ced850f2fb68f3555c2b1d224b9a356e36e1865bfaa7: Status 404 returned error can't find the container with id fef79d49a6eedf266893ced850f2fb68f3555c2b1d224b9a356e36e1865bfaa7
Oct 30 22:00:08.991620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.991580    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" event={"ID":"1d64be76-3b79-413a-869d-4d9021a06e87","Type":"ContainerStarted","Data":"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"}
Oct 30 22:00:08.992068 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.991682    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:08.993146 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.993124    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" event={"ID":"680bf5c9-2b25-407f-8a86-896031ab5184","Type":"ContainerStarted","Data":"fef79d49a6eedf266893ced850f2fb68f3555c2b1d224b9a356e36e1865bfaa7"}
Oct 30 22:00:08.995217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.995189    4295 generic.go:356] "Generic (PLEG): container finished" podID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerID="c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4" exitCode=60
Oct 30 22:00:08.995358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.995275    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerDied","Data":"c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4"}
Oct 30 22:00:08.995358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.995340    4295 scope.go:122] "RemoveContainer" containerID="151feb27b808953e30c4a4f6c7d715e37a9e8be90d379a42f9e06c88bb2f208c"
Oct 30 22:00:08.995891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:08.995871    4295 scope.go:122] "RemoveContainer" containerID="c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4"
Oct 30 22:00:08.996117 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:08.996096    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b_odh-ai-gateway-infra(b5c1032d-5e0a-475f-9eb7-43f0402942c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9"
Oct 30 22:00:09.117078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:09.116954    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" podStartSLOduration=1.72064005 podStartE2EDuration="4.116934118s" totalImagesPullingTime="2.396294068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.116769237 +0000 UTC m=+611.759363027" watchObservedRunningTime="2026-10-30 22:00:09.116934118 +0000 UTC m=+611.759527907"
Oct 30 22:00:10.002073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:10.002027    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" event={"ID":"680bf5c9-2b25-407f-8a86-896031ab5184","Type":"ContainerStarted","Data":"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"}
Oct 30 22:00:10.002558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:10.002140    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:10.004019 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:10.003997    4295 scope.go:122] "RemoveContainer" containerID="c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4"
Oct 30 22:00:10.004298 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:10.004274    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b_odh-ai-gateway-infra(b5c1032d-5e0a-475f-9eb7-43f0402942c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9"
Oct 30 22:00:10.140140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:10.140084    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" podStartSLOduration=2.140069873 podStartE2EDuration="2.140069873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:10.139607783 +0000 UTC m=+612.782201574" watchObservedRunningTime="2026-10-30 22:00:10.140069873 +0000 UTC m=+612.782663662"
Oct 30 22:00:12.014819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.014788    4295 generic.go:356] "Generic (PLEG): container finished" podID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerID="4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2" exitCode=0
Oct 30 22:00:12.015278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.014859    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerDied","Data":"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"}
Oct 30 22:00:12.015278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.015219    4295 scope.go:122] "RemoveContainer" containerID="4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"
Oct 30 22:00:12.016559 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.016530    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:12.016620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.016597    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:12.016654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:12.016631    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:13.038837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:13.038791    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerStarted","Data":"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"}
Oct 30 22:00:13.534689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:13.533711    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:00:13.538944 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:13.538795    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 22:00:13.538944 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:13.538872    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 22:00:14.531149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:14.531114    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 22:00:14.531781 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:14.531735    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 22:00:14.820742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:14.819762    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:00:15.537900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:15.537868    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 22:00:15.538524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:15.538476    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 22:00:16.530050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:16.530011    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 22:00:16.530347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:16.530320    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 22:00:17.551581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:17.551538    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 22:00:17.556521 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:17.554796    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 22:00:18.532929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:18.532901    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:00:18.534159 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:18.534132    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:00:18.999322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:18.999291    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:00:18.999751 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:18.999380    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:00:20.006928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:20.006899    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:00:21.012168 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:21.012141    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:00:22.083050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.083017    4295 generic.go:356] "Generic (PLEG): container finished" podID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerID="23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3" exitCode=0
Oct 30 22:00:22.083469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.083094    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerDied","Data":"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"}
Oct 30 22:00:22.083469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.083442    4295 scope.go:122] "RemoveContainer" containerID="23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"
Oct 30 22:00:22.084690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.084643    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:22.084764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.084732    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:22.084799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.084764    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:22.486621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:22.486596    4295 scope.go:122] "RemoveContainer" containerID="c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4"
Oct 30 22:00:23.096687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.089422    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerStarted","Data":"d6b9a889eb8644820e7a0a98b2b65802489dc054bcb76674237403b0b434735f"}
Oct 30 22:00:23.115347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.115315    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerStarted","Data":"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"}
Oct 30 22:00:23.135619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.135591    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:00:23.139018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.139002    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.209688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.201971    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-2182a5-serving-cert\""
Oct 30 22:00:23.215019 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.214979    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:00:23.215374 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.215182    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 22:00:23.215493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.215405    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 22:00:23.230863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.230806    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" podStartSLOduration=17.354046447 podStartE2EDuration="23.230790188s" totalImagesPullingTime="5.876743741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:23.215501177 +0000 UTC m=+625.858094967" watchObservedRunningTime="2026-10-30 22:00:23.230790188 +0000 UTC m=+625.873383981"
Oct 30 22:00:23.232169 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.232144    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:00:23.234443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.234416    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdps4\" (UniqueName: \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.234579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.234559    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.234691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.234649    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.234758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.234715    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.335410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.335370    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdps4\" (UniqueName: \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.335605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.335457    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.335605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.335495    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.335605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.335527    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.335804 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:23.335717    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-serving-cert: secret "maas-api-e2e-worker-w6-2182a5-serving-cert" not found
Oct 30 22:00:23.335804 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:23.335793    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls podName:c1717471-16bd-496b-b94a-1cb41698b09c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:23.835771271 +0000 UTC m=+626.478365052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls") pod "maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" (UID: "c1717471-16bd-496b-b94a-1cb41698b09c") : secret "maas-api-e2e-worker-w6-2182a5-serving-cert" not found
Oct 30 22:00:23.336111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.336092    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.338092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.338061    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.414794 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.414753    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdps4\" (UniqueName: \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.631648 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:23.631614    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:00:23.841566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.841485    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:23.844036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:23.844006    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:24.051623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.051589    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:24.122814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.122730    4295 generic.go:356] "Generic (PLEG): container finished" podID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerID="d6b9a889eb8644820e7a0a98b2b65802489dc054bcb76674237403b0b434735f" exitCode=60
Oct 30 22:00:24.123248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.122802    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerDied","Data":"d6b9a889eb8644820e7a0a98b2b65802489dc054bcb76674237403b0b434735f"}
Oct 30 22:00:24.123248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.122869    4295 scope.go:122] "RemoveContainer" containerID="c1df6f75de4b52b3df4d4ae30a5a3472adc208811fc89064bbc4c189dc26b7f4"
Oct 30 22:00:24.123248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.123210    4295 scope.go:122] "RemoveContainer" containerID="d6b9a889eb8644820e7a0a98b2b65802489dc054bcb76674237403b0b434735f"
Oct 30 22:00:24.123545 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:24.123479    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b_odh-ai-gateway-infra(b5c1032d-5e0a-475f-9eb7-43f0402942c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9"
Oct 30 22:00:24.212998 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.212967    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 22:00:24.213196 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.213011    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 22:00:24.231717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.231685    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:24.463743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:24.463706    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:00:24.471403 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:24.471371    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1717471_16bd_496b_b94a_1cb41698b09c.slice/crio-ccf2bcdb114c6d0646a2e10791242fe1a7bd93409bf0abef19487a78616c7b0e WatchSource:0}: Error finding container ccf2bcdb114c6d0646a2e10791242fe1a7bd93409bf0abef19487a78616c7b0e: Status 404 returned error can't find the container with id ccf2bcdb114c6d0646a2e10791242fe1a7bd93409bf0abef19487a78616c7b0e
Oct 30 22:00:25.130733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.130700    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" event={"ID":"c1717471-16bd-496b-b94a-1cb41698b09c","Type":"ContainerStarted","Data":"ccf2bcdb114c6d0646a2e10791242fe1a7bd93409bf0abef19487a78616c7b0e"}
Oct 30 22:00:25.216240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.216180    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 22:00:25.216539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.216504    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 22:00:25.273004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.272981    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:25.358316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.358289    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\") pod \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") "
Oct 30 22:00:25.358411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.358394    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\" (UniqueName: \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\") pod \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\" (UID: \"b5c1032d-5e0a-475f-9eb7-43f0402942c9\") "
Oct 30 22:00:25.358615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.358593    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca" pod "b5c1032d-5e0a-475f-9eb7-43f0402942c9" (UID: "b5c1032d-5e0a-475f-9eb7-43f0402942c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:25.360711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.360684    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj" pod "b5c1032d-5e0a-475f-9eb7-43f0402942c9" (UID: "b5c1032d-5e0a-475f-9eb7-43f0402942c9"). InnerVolumeSpecName "kube-api-access-xnqkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:25.459286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.459258    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c1032d-5e0a-475f-9eb7-43f0402942c9-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:25.459286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:25.459284    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqkj\" (UniqueName: \"kubernetes.io/projected/b5c1032d-5e0a-475f-9eb7-43f0402942c9-kube-api-access-xnqkj\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:26.136707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.136644    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b" event={"ID":"b5c1032d-5e0a-475f-9eb7-43f0402942c9","Type":"ContainerDied","Data":"5b69276f6d943a8525550823c1bf5840f483988b292199c85f6c3a3cdf7c34f0"}
Oct 30 22:00:26.136707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.136689    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"
Oct 30 22:00:26.137247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.136728    4295 scope.go:122] "RemoveContainer" containerID="d6b9a889eb8644820e7a0a98b2b65802489dc054bcb76674237403b0b434735f"
Oct 30 22:00:26.138862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.138764    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" event={"ID":"c1717471-16bd-496b-b94a-1cb41698b09c","Type":"ContainerStarted","Data":"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"}
Oct 30 22:00:26.138970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.138920    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:26.213066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.213033    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 22:00:26.213209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.213099    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 22:00:26.482216 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.482161    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" podStartSLOduration=2.604018368 podStartE2EDuration="3.482147154s" totalImagesPullingTime="878.128786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:26.356117577 +0000 UTC m=+628.998711368" watchObservedRunningTime="2026-10-30 22:00:26.482147154 +0000 UTC m=+629.124740942"
Oct 30 22:00:26.483476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.483454    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:26.486804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:26.486780    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889960-lwq4b"]
Oct 30 22:00:27.213116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:27.213074    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 22:00:27.213506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:27.213142    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 22:00:27.514538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:27.514437    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" path="/var/lib/kubelet/pods/b5c1032d-5e0a-475f-9eb7-43f0402942c9/volumes"
Oct 30 22:00:28.216260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:28.216223    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:00:28.216752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:28.216647    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:00:28.217476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:28.217457    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:00:32.150003 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:32.149977    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:00:40.071140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071105    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:00:40.071520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071467    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.071520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071477    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.071520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071497    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.071520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071502    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.071647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071567    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.071647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.071576    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:00:40.076037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.076017    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.130342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.130307    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-bce513-serving-cert\""
Oct 30 22:00:40.181778 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.181742    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:00:40.223235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.223192    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.223442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.223313    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.223442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.223364    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds472\" (UniqueName: \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.223442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.223389    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.324601 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.324502    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.324601 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.324589    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.324887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.324646    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds472\" (UniqueName: \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.324887 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:40.324693    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-serving-cert: secret "maas-api-e2e-worker-w4-bce513-serving-cert" not found
Oct 30 22:00:40.324887 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:40.324779    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls podName:1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea nodeName:}" failed. No retries permitted until 2026-10-30 22:00:40.824757974 +0000 UTC m=+643.467351754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls") pod "maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" (UID: "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea") : secret "maas-api-e2e-worker-w4-bce513-serving-cert" not found
Oct 30 22:00:40.324887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.324862    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.325255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.325236    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.327463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.327442    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.414532 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.414494    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds472\" (UniqueName: \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.829542 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.829499    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.832170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.832143    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:40.996213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:40.994864    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:41.413774 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:41.405652    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:00:41.415374 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:00:41.415332    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea54f7f_4cd0_4d07_b822_ce9603f8a4ea.slice/crio-189bb5e9dcd1b1a659bce76853458944879567b91f21f52d419f929f45c9d4e7 WatchSource:0}: Error finding container 189bb5e9dcd1b1a659bce76853458944879567b91f21f52d419f929f45c9d4e7: Status 404 returned error can't find the container with id 189bb5e9dcd1b1a659bce76853458944879567b91f21f52d419f929f45c9d4e7
Oct 30 22:00:42.234122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:42.234093    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" event={"ID":"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea","Type":"ContainerStarted","Data":"189bb5e9dcd1b1a659bce76853458944879567b91f21f52d419f929f45c9d4e7"}
Oct 30 22:00:43.241617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:43.241575    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" event={"ID":"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea","Type":"ContainerStarted","Data":"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"}
Oct 30 22:00:43.242101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:43.241638    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:43.374852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:43.374796    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" podStartSLOduration=2.428002946 podStartE2EDuration="3.374782089s" totalImagesPullingTime="946.779143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:43.373883765 +0000 UTC m=+646.016477555" watchObservedRunningTime="2026-10-30 22:00:43.374782089 +0000 UTC m=+646.017375877"
Oct 30 22:00:46.259342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.259305    4295 generic.go:356] "Generic (PLEG): container finished" podID="d6736d70-17a0-4076-9119-6fd677801fe1" containerID="abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd" exitCode=0
Oct 30 22:00:46.259785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.259349    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerDied","Data":"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"}
Oct 30 22:00:46.259785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.259679    4295 scope.go:122] "RemoveContainer" containerID="abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"
Oct 30 22:00:46.260867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.260838    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:46.260928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.260905    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:46.260962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.260937    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:46.825956 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:46.825906    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:00:47.267486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:47.267442    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerStarted","Data":"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"}
Oct 30 22:00:47.887099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:47.887071    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:00:47.902697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:47.901831    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 22:00:47.902697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:47.901914    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 22:00:48.887297 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:48.887249    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 22:00:48.887816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:48.887314    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 22:00:49.252354 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:49.252325    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:00:49.887054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:49.887018    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 22:00:49.887234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:49.887082    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 22:00:50.887337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:50.887296    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 22:00:50.887790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:50.887364    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 22:00:51.886410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:51.886369    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 22:00:51.886620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:51.886432    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 22:00:52.891314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:52.891283    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:00:52.891756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:52.891559    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:00:52.892340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:00:52.892325    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:00:53.608332 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:00:53.608290    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:01:19.538622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.538539    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:01:19.539078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.538969    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:01:19.539078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.538981    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:01:19.539078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.539068    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c1032d-5e0a-475f-9eb7-43f0402942c9" containerName="cleanup"
Oct 30 22:01:19.541077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.541062    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.615309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.615273    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"kube-root-ca.crt\""
Oct 30 22:01:19.645012 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.644986    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"openshift-service-ca.crt\""
Oct 30 22:01:19.645141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.645113    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"default-dockercfg-f8swn\""
Oct 30 22:01:19.645182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.645174    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"simulator-w5-3783c2-kserve-self-signed-certs\""
Oct 30 22:01:19.647743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.647727    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:01:19.651386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651367    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.651436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651409    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.651471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651433    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.651504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651478    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cxf\" (UniqueName: \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.651569 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651550    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.651627 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.651613    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.752978    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753027    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753060    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753083    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cxf\" (UniqueName: \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753115    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753184    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753554    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753651 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753602    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.753756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.753740    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.755478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.755455    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.755655 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.755639    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.817075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.817019    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cxf\" (UniqueName: \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\") pod \"simulator-w5-3783c2-kserve-fbc59d5db-b4jct\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") " pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:19.851193 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:19.851162    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:20.227263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:20.227240    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:01:20.229464 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:01:20.229428    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f23d29b_ea29_4dc6_8d97_a7b4bf85083d.slice/crio-5502d4fd7f662ba3ee9978852a5b15514a957873405f90cfe9298a1239f2315f WatchSource:0}: Error finding container 5502d4fd7f662ba3ee9978852a5b15514a957873405f90cfe9298a1239f2315f: Status 404 returned error can't find the container with id 5502d4fd7f662ba3ee9978852a5b15514a957873405f90cfe9298a1239f2315f
Oct 30 22:01:20.420745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:20.420709    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" event={"ID":"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d","Type":"ContainerStarted","Data":"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"}
Oct 30 22:01:20.420897 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:20.420759    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" event={"ID":"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d","Type":"ContainerStarted","Data":"5502d4fd7f662ba3ee9978852a5b15514a957873405f90cfe9298a1239f2315f"}
Oct 30 22:01:20.543838 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:20.543746    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podStartSLOduration=1.5437329549999999 podStartE2EDuration="1.543732955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:20.542422061 +0000 UTC m=+683.185015842" watchObservedRunningTime="2026-10-30 22:01:20.543732955 +0000 UTC m=+683.186326744"
Oct 30 22:01:20.852183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:20.852113    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:23.609767 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:01:23.609730    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:01:29.851303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:29.851271    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:29.864107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:29.864085    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:30.477599 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:30.477573    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:01:44.984529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:44.984496    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:01:44.987945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:44.987929    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.018920 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.018891    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.019085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.018943    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.019085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.018962    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x889b\" (UniqueName: \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.019171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.019111    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.019202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.019189    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.019286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.019271    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.092573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.092530    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"default-dockercfg-47zx6\""
Oct 30 22:01:45.092771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.092645    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"premium-w2-d8837a-kserve-self-signed-certs\""
Oct 30 22:01:45.092771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.092646    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"openshift-service-ca.crt\""
Oct 30 22:01:45.092771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.092692    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"kube-root-ca.crt\""
Oct 30 22:01:45.095913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.095895    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:01:45.140334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.137541    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.140722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.140688    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.141962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.140942    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.148451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.147801    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x889b\" (UniqueName: \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.148451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.147875    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.148451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.147961    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.151134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.147138    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.151134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.147507    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.151134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.150039    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.156611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.156484    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.157475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.156700    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.216497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.216465    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x889b\" (UniqueName: \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\") pod \"premium-w2-d8837a-kserve-5b647787c5-m2f62\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") " pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.299756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.299679    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:45.694784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:45.694762    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:01:45.696509 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:01:45.696487    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17e2a475_5742_4f29_9572_e8d2ec726ce2.slice/crio-00f1842bfa3a6a5eb833c350c98ea8fb08c90f32debaa289785168c0328a33c8 WatchSource:0}: Error finding container 00f1842bfa3a6a5eb833c350c98ea8fb08c90f32debaa289785168c0328a33c8: Status 404 returned error can't find the container with id 00f1842bfa3a6a5eb833c350c98ea8fb08c90f32debaa289785168c0328a33c8
Oct 30 22:01:46.552799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:46.552762    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" event={"ID":"17e2a475-5742-4f29-9572-e8d2ec726ce2","Type":"ContainerStarted","Data":"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"}
Oct 30 22:01:46.553192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:46.552810    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" event={"ID":"17e2a475-5742-4f29-9572-e8d2ec726ce2","Type":"ContainerStarted","Data":"00f1842bfa3a6a5eb833c350c98ea8fb08c90f32debaa289785168c0328a33c8"}
Oct 30 22:01:46.676823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:46.676775    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podStartSLOduration=2.676760878 podStartE2EDuration="2.676760878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.674473379 +0000 UTC m=+709.317067169" watchObservedRunningTime="2026-10-30 22:01:46.676760878 +0000 UTC m=+709.319354668"
Oct 30 22:01:47.300021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:47.299990    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:53.606699 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:01:53.606641    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:01:55.300609 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:55.300572    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:55.313090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:55.313061    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:01:55.613965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:01:55.613863    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:02:23.610430 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:02:23.610390    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:02:28.472269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.472224    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:02:28.475879 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.475860    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.529805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.529778    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"openshift-service-ca.crt\""
Oct 30 22:02:28.529953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.529909    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"kube-root-ca.crt\""
Oct 30 22:02:28.580579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.580544    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"default-dockercfg-7cw79\""
Oct 30 22:02:28.580579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.580544    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"premium-w3-8ca52a-kserve-self-signed-certs\""
Oct 30 22:02:28.582397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.582379    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:02:28.605061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605038    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.605207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605072    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.605207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605094    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.605465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605438    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.605576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605520    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.605647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.605585    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxrl\" (UniqueName: \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.706738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.706704    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.706909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.706758    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.706909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.706874    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxrl\" (UniqueName: \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.706975    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.707006    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.707042    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.707057    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.707332    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.707495 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.707476    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.709135 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.709118    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.709501 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.709485    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.770005 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.769947    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxrl\" (UniqueName: \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\") pod \"premium-w3-8ca52a-kserve-586dc77669-pfqbz\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") " pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:28.788850 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:28.788827    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:29.191365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:29.191338    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:02:29.193017 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:02:29.192992    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a824055_9e2c_46df_ae43_ae9c0bd169fd.slice/crio-abf0ede6eee5f47d1430e1c82c88ba8c29a6ef4177c61220deeba28469d94a2c WatchSource:0}: Error finding container abf0ede6eee5f47d1430e1c82c88ba8c29a6ef4177c61220deeba28469d94a2c: Status 404 returned error can't find the container with id abf0ede6eee5f47d1430e1c82c88ba8c29a6ef4177c61220deeba28469d94a2c
Oct 30 22:02:29.760449 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:29.760416    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" event={"ID":"8a824055-9e2c-46df-ae43-ae9c0bd169fd","Type":"ContainerStarted","Data":"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"}
Oct 30 22:02:29.760837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:29.760462    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" event={"ID":"8a824055-9e2c-46df-ae43-ae9c0bd169fd","Type":"ContainerStarted","Data":"abf0ede6eee5f47d1430e1c82c88ba8c29a6ef4177c61220deeba28469d94a2c"}
Oct 30 22:02:29.789747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:29.789717    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:29.881783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:29.881736    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podStartSLOduration=1.881723123 podStartE2EDuration="1.881723123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:29.880825626 +0000 UTC m=+752.523419416" watchObservedRunningTime="2026-10-30 22:02:29.881723123 +0000 UTC m=+752.524316939"
Oct 30 22:02:38.789415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:38.789380    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:38.802523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:38.802491    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:38.815613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:02:38.815589    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:02:53.608875 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:02:53.608842    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:03:11.764955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.764921    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:03:11.768288 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.768271    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.842254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.842225    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"premium-w0-6a0a09-kserve-self-signed-certs\""
Oct 30 22:03:11.842419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.842310    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"openshift-service-ca.crt\""
Oct 30 22:03:11.848894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.848869    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.849040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.848909    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.849040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.848939    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.849171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.849072    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.849171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.849144    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.849275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.849176    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss8f\" (UniqueName: \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.873161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.873143    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"default-dockercfg-k92f6\""
Oct 30 22:03:11.873451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.873433    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"kube-root-ca.crt\""
Oct 30 22:03:11.875799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.875780    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:03:11.949844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949808    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949860    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949889    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949920    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949942    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.949959    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss8f\" (UniqueName: \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.950271    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.950300    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.950359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.950347    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.952416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.952380    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:11.952676 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:11.952648    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:12.013244 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:12.013213    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss8f\" (UniqueName: \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\") pod \"premium-w0-6a0a09-kserve-8694c89769-58vgn\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") " pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:12.079687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:12.079578    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:12.488471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:12.488444    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:03:12.490880 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:03:12.490854    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1619a3eb_e341_416c_b31d_95d099322b62.slice/crio-6865c0856a156007d8b02e6d38b0977fe933025eabde6ebbcb2a02d910f0cba7 WatchSource:0}: Error finding container 6865c0856a156007d8b02e6d38b0977fe933025eabde6ebbcb2a02d910f0cba7: Status 404 returned error can't find the container with id 6865c0856a156007d8b02e6d38b0977fe933025eabde6ebbcb2a02d910f0cba7
Oct 30 22:03:12.970059 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:12.970019    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" event={"ID":"1619a3eb-e341-416c-b31d-95d099322b62","Type":"ContainerStarted","Data":"46d1374f588cc778044f867944d8c1929f0f3e4c8dfd262c68a1506ecf9f2cf4"}
Oct 30 22:03:12.970059 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:12.970059    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" event={"ID":"1619a3eb-e341-416c-b31d-95d099322b62","Type":"ContainerStarted","Data":"6865c0856a156007d8b02e6d38b0977fe933025eabde6ebbcb2a02d910f0cba7"}
Oct 30 22:03:13.079875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:13.079845    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:13.092419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:13.092368    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podStartSLOduration=2.092355493 podStartE2EDuration="2.092355493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:13.091871712 +0000 UTC m=+795.734465513" watchObservedRunningTime="2026-10-30 22:03:13.092355493 +0000 UTC m=+795.734949337"
Oct 30 22:03:22.079983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:22.079949    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:22.092415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:22.092391    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:23.039026 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:23.038995    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:03:23.608948 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:03:23.608910    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:03:36.129530 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.129500    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:03:36.133139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.133123    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.187709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.187683    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"openshift-service-ca.crt\""
Oct 30 22:03:36.200218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200188    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.200400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200249    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jjq\" (UniqueName: \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.200400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200331    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.200400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200363    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.200400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200389    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.200568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.200441    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.237878 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.237857    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"premium-w1-9cff60-kserve-self-signed-certs\""
Oct 30 22:03:36.237998 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.237857    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"kube-root-ca.crt\""
Oct 30 22:03:36.237998 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.237857    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"default-dockercfg-gwrpm\""
Oct 30 22:03:36.240127 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.240109    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:03:36.301761 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301735    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.301914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301774    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jjq\" (UniqueName: \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.301914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301795    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.301914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301815    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.301914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301848    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.301914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.301871    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.302245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.302226    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.302313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.302276    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.302313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.302293    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.304155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.304138    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.304426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.304407    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.365749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.365720    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jjq\" (UniqueName: \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\") pod \"premium-w1-9cff60-kserve-558d6dc6dc-vbvm6\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") " pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.445341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.445305    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:36.842924 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:36.842900    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:03:36.844698 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:03:36.844634    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod157cd977_670d_41d5_af48_0b9005280f57.slice/crio-cec61e348bf2e317310cae26f89fa95c264258e9596c64bf4db01de26dd81e3d WatchSource:0}: Error finding container cec61e348bf2e317310cae26f89fa95c264258e9596c64bf4db01de26dd81e3d: Status 404 returned error can't find the container with id cec61e348bf2e317310cae26f89fa95c264258e9596c64bf4db01de26dd81e3d
Oct 30 22:03:37.105261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:37.105173    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" event={"ID":"157cd977-670d-41d5-af48-0b9005280f57","Type":"ContainerStarted","Data":"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"}
Oct 30 22:03:37.105261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:37.105216    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" event={"ID":"157cd977-670d-41d5-af48-0b9005280f57","Type":"ContainerStarted","Data":"cec61e348bf2e317310cae26f89fa95c264258e9596c64bf4db01de26dd81e3d"}
Oct 30 22:03:37.227467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:37.227410    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podStartSLOduration=1.227395485 podStartE2EDuration="1.227395485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:37.225626058 +0000 UTC m=+819.868219846" watchObservedRunningTime="2026-10-30 22:03:37.227395485 +0000 UTC m=+819.869989274"
Oct 30 22:03:37.445465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:37.445433    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:46.446455 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:46.446423    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:46.459438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:46.459415    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:47.173077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:03:47.173042    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:03:53.611596 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:03:53.611550    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:04:05.299755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:05.299717    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 22:04:05.300369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:05.299952    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy" containerID="cri-o://dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d" gracePeriod=30
Oct 30 22:04:10.562116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.562092    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 22:04:10.666561 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666480    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.666561 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666546    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.666807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666592    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.666807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666655    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.666807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666742    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.666807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666804    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666800    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666850    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666889    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666935    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\") pod \"a1001cfb-75b3-49cb-89d1-d14d2333828d\" (UID: \"a1001cfb-75b3-49cb-89d1-d14d2333828d\") "
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666960    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.667028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.666970    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.667355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667160    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.667355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667303    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.667355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667316    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.667355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667338    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.667355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667352    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.667536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.667360    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.668953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.668929    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:10.669341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.669323    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.669553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.669533    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.669553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.669539    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz" pod "a1001cfb-75b3-49cb-89d1-d14d2333828d" (UID: "a1001cfb-75b3-49cb-89d1-d14d2333828d"). InnerVolumeSpecName "kube-api-access-pr2mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.768619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.768586    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.768619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.768620    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.768845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.768634    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1001cfb-75b3-49cb-89d1-d14d2333828d-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.768845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.768647    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2mz\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-kube-api-access-pr2mz\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.768845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:10.768679    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1001cfb-75b3-49cb-89d1-d14d2333828d-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.280678 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.280641    4295 generic.go:356] "Generic (PLEG): container finished" podID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerID="dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d" exitCode=0
Oct 30 22:04:11.280848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.280714    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"
Oct 30 22:04:11.280848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.280724    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" event={"ID":"a1001cfb-75b3-49cb-89d1-d14d2333828d","Type":"ContainerDied","Data":"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"}
Oct 30 22:04:11.280848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.280778    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5" event={"ID":"a1001cfb-75b3-49cb-89d1-d14d2333828d","Type":"ContainerDied","Data":"8a5ab51a0ab3fe8a36c7af4e7e839118a2fbd1f96a15e1a865a6bd9242b1a959"}
Oct 30 22:04:11.280848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.280800    4295 scope.go:122] "RemoveContainer" containerID="dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"
Oct 30 22:04:11.290253 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.290234    4295 scope.go:122] "RemoveContainer" containerID="dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"
Oct 30 22:04:11.290536 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:04:11.290515    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d\": container with ID starting with dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d not found: ID does not exist" containerID="dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"
Oct 30 22:04:11.290580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.290543    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d"} err="failed to get container status \"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d\": rpc error: code = NotFound desc = could not find container \"dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d\": container with ID starting with dec596764000cd14ed49e8d7d4b058baba2839d19450f43f8cc726560769005d not found: ID does not exist"
Oct 30 22:04:11.501478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.501447    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 22:04:11.551892 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:11.551822    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5"]
Oct 30 22:04:13.491952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:13.491921    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" path="/var/lib/kubelet/pods/a1001cfb-75b3-49cb-89d1-d14d2333828d/volumes"
Oct 30 22:04:15.603966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.603935    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:04:15.604343 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.604330    4295 cpu_manager.go:414] "containerMap: removing container" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy"
Oct 30 22:04:15.604396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.604344    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy"
Oct 30 22:04:15.604431 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.604425    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1001cfb-75b3-49cb-89d1-d14d2333828d" containerName="istio-proxy"
Oct 30 22:04:15.607497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.607479    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.662401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.662373    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bce513-openshift-default-dockercfg-jq86d\""
Oct 30 22:04:15.714154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.714127    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:04:15.720372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720349    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720391    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720420    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720491    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720529    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720565    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720583    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720673    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5g2\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.720813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.720763    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822108    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822161    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822189    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822206    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822224    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822243    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822293    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822599 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822317    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822599 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822379    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5g2\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822646    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822726    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822813    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.822940 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.822913    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.823237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.823220    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.824847 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.824819    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.824996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.824975    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.885729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.885650    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.891912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.891893    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5g2\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:15.922878 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:15.922852    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:16.356255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:16.356230    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:04:16.357954 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:04:16.357929    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b17a3d_f7c9_4d07_be77_75eb94f623fc.slice/crio-47a7c79e938bafcc3d7a0d1ea3036fa750003f412bcd7771462a6d72bb3af3b0 WatchSource:0}: Error finding container 47a7c79e938bafcc3d7a0d1ea3036fa750003f412bcd7771462a6d72bb3af3b0: Status 404 returned error can't find the container with id 47a7c79e938bafcc3d7a0d1ea3036fa750003f412bcd7771462a6d72bb3af3b0
Oct 30 22:04:16.360273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:16.360243    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:16.360330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:16.360308    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:16.360365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:16.360336    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:17.317683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.317635    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" event={"ID":"c0b17a3d-f7c9-4d07-be77-75eb94f623fc","Type":"ContainerStarted","Data":"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"}
Oct 30 22:04:17.318096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.317719    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" event={"ID":"c0b17a3d-f7c9-4d07-be77-75eb94f623fc","Type":"ContainerStarted","Data":"47a7c79e938bafcc3d7a0d1ea3036fa750003f412bcd7771462a6d72bb3af3b0"}
Oct 30 22:04:17.503690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.497355    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podStartSLOduration=2.497334799 podStartE2EDuration="2.497334799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:17.49595575 +0000 UTC m=+860.138549540" watchObservedRunningTime="2026-10-30 22:04:17.497334799 +0000 UTC m=+860.139928589"
Oct 30 22:04:17.923505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.923462    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:17.925010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.924980    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.925111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:17.925044    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 22:04:18.923614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:18.923579    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.923997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:18.923642    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 22:04:19.923408 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:19.923355    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.923590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:19.923426    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 22:04:20.923916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:20.923880    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.924347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:20.923961    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 22:04:21.923365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:21.923330    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.923544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:21.923393    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 22:04:23.138074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:23.138041    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:23.138899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:23.138872    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:23.139995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:04:23.139971    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:04:23.609419 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:04:23.609383    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:04:53.608612 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:04:53.608578    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:05:05.820276 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.820243    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:05:05.825057 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.825040    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.874779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.874747    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:05:05.878240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.878225    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.925095 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.925056    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:05:05.946255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946216    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946271    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946301    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946392    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946512 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946417    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946512 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946495    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946516    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946539    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946558    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946622    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946740    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946782 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946761    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946783    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.946818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946804    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946906    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.946967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.946952    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.947018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.947004    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697gr\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:05.947052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.947031    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqwg\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:05.976720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:05.976688    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:05:06.048258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048218    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048460 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048268    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.048460 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048445    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697gr\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048488    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqwg\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.048589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048535    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048564    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048592    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048636    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048686    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048718    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048732    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.048801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048782    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048820    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048852    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048888    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048926    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.048985    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049033    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049037    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049043    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049067    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049073    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049098    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049336    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049619    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.049685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049626    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049633    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.049831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.049750    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.051302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.051270    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.051780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.051758    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.051860 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.051815    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.052032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.052013    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.118389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.118324    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697gr\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.120823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.120800    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.138834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.138812    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqwg\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\") pod \"maas-default-gateway-openshift-default-67bf986f99-f9slk\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.142432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.142407    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hdhvg\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.190581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.190555    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:06.438218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.438190    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:06.620372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.620349    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:05:06.621777 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:05:06.621752    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1f297_079c_4d20_8a1b_12e45840ab9e.slice/crio-57aa4053ad9c0e2077d1a9659233c9037f4f7b3aa4fccdd8195e89e417cf2c4b WatchSource:0}: Error finding container 57aa4053ad9c0e2077d1a9659233c9037f4f7b3aa4fccdd8195e89e417cf2c4b: Status 404 returned error can't find the container with id 57aa4053ad9c0e2077d1a9659233c9037f4f7b3aa4fccdd8195e89e417cf2c4b
Oct 30 22:05:06.624003 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.623972    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.624072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.624034    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.624072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.624061    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.820192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.820073    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:05:06.822544 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:05:06.822518    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472d6bf4_6c02_44ca_97de_27d02cdd1035.slice/crio-f12500ca010bb7e66549dd72d0290e50479ca25423f43974fd437c1448d96ede WatchSource:0}: Error finding container f12500ca010bb7e66549dd72d0290e50479ca25423f43974fd437c1448d96ede: Status 404 returned error can't find the container with id f12500ca010bb7e66549dd72d0290e50479ca25423f43974fd437c1448d96ede
Oct 30 22:05:06.825095 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.825063    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.825179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.825135    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.825179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:06.825164    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:07.642813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.642774    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" event={"ID":"472d6bf4-6c02-44ca-97de-27d02cdd1035","Type":"ContainerStarted","Data":"f463d3e326b0a52bc7eb9ed8f89135eda437d078ef37e18f188c069af619eb7d"}
Oct 30 22:05:07.643118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.643084    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" event={"ID":"472d6bf4-6c02-44ca-97de-27d02cdd1035","Type":"ContainerStarted","Data":"f12500ca010bb7e66549dd72d0290e50479ca25423f43974fd437c1448d96ede"}
Oct 30 22:05:07.645141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.645113    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" event={"ID":"34b1f297-079c-4d20-8a1b-12e45840ab9e","Type":"ContainerStarted","Data":"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"}
Oct 30 22:05:07.645274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.645160    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" event={"ID":"34b1f297-079c-4d20-8a1b-12e45840ab9e","Type":"ContainerStarted","Data":"57aa4053ad9c0e2077d1a9659233c9037f4f7b3aa4fccdd8195e89e417cf2c4b"}
Oct 30 22:05:07.823687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.823135    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podStartSLOduration=2.823118198 podStartE2EDuration="2.823118198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:07.822190357 +0000 UTC m=+910.464784147" watchObservedRunningTime="2026-10-30 22:05:07.823118198 +0000 UTC m=+910.465711988"
Oct 30 22:05:07.945454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:07.945399    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podStartSLOduration=2.945384084 podStartE2EDuration="2.945384084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:07.945065336 +0000 UTC m=+910.587659126" watchObservedRunningTime="2026-10-30 22:05:07.945384084 +0000 UTC m=+910.587977872"
Oct 30 22:05:08.195746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.193415    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:08.195746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.193577    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:08.195746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.193635    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:08.439689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.439395    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:08.441099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.440963    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:08.441099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:08.441029    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:09.201688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:09.192024    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:09.201688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:09.192086    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:09.439691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:09.439084    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:09.439691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:09.439153    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:10.214696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:10.206602    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:10.214696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:10.206688    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:10.442721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:10.439654    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:10.442721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:10.440687    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:11.192030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.191991    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:11.192316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.192286    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:11.439114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.439074    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:11.439722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.439689    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:11.534600 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.534498    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"]
Oct 30 22:05:11.539596 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.539563    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.600691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.598406    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:11.600691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.598492    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:11.631693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.627384    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"]
Oct 30 22:05:11.668180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.668147    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.668359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.668205    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2rz\" (UniqueName: \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.668359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.668251    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.769623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.769577    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.769833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.769634    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2rz\" (UniqueName: \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.769833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.769688    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.770278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.770256    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.772150 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.772129    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.843151 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.843071    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2rz\" (UniqueName: \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\") pod \"maas-api-revoke-sub-d5d72420-9lkj5\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:11.854229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:11.854198    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:12.213993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.213474    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:12.213993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.213538    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:12.277068 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.277035    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"]
Oct 30 22:05:12.278790 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:05:12.278759    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe5eb62_a1f5_48a8_85d3_9248e775687a.slice/crio-c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304 WatchSource:0}: Error finding container c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304: Status 404 returned error can't find the container with id c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304
Oct 30 22:05:12.448232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.448089    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:12.448232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.448158    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:12.675761 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:12.675726    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5" event={"ID":"bbe5eb62-a1f5-48a8-85d3-9248e775687a","Type":"ContainerStarted","Data":"c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304"}
Oct 30 22:05:13.191589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.191548    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:13.191822 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.191606    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:13.439340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.439299    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:13.439506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.439387    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:13.683727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.683691    4295 generic.go:356] "Generic (PLEG): container finished" podID="bbe5eb62-a1f5-48a8-85d3-9248e775687a" containerID="9eb2296dcf1e5b80fb5b10ed4ef5a7ba3734b7ab218d05887fb5be58af30fd71" exitCode=0
Oct 30 22:05:13.684133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:13.683761    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5" event={"ID":"bbe5eb62-a1f5-48a8-85d3-9248e775687a","Type":"ContainerDied","Data":"9eb2296dcf1e5b80fb5b10ed4ef5a7ba3734b7ab218d05887fb5be58af30fd71"}
Oct 30 22:05:14.191145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:14.191106    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f9slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:05:14.191314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:14.191188    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:05:14.439188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:14.439140    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:05:14.439364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:14.439210    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:05:14.893875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:14.893832    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:15.040536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.040024    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\") pod \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") "
Oct 30 22:05:15.040869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.040845    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\" (UniqueName: \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\") pod \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") "
Oct 30 22:05:15.041087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.041067    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\") pod \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\" (UID: \"bbe5eb62-a1f5-48a8-85d3-9248e775687a\") "
Oct 30 22:05:15.041755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.040459    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca" pod "bbe5eb62-a1f5-48a8-85d3-9248e775687a" (UID: "bbe5eb62-a1f5-48a8-85d3-9248e775687a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:15.065757 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.065398    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca" pod "bbe5eb62-a1f5-48a8-85d3-9248e775687a" (UID: "bbe5eb62-a1f5-48a8-85d3-9248e775687a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:15.076213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.076146    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz" pod "bbe5eb62-a1f5-48a8-85d3-9248e775687a" (UID: "bbe5eb62-a1f5-48a8-85d3-9248e775687a"). InnerVolumeSpecName "kube-api-access-6d2rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:15.142907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.142869    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:15.142907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.142909    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5eb62-a1f5-48a8-85d3-9248e775687a-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:15.143176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.142926    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2rz\" (UniqueName: \"kubernetes.io/projected/bbe5eb62-a1f5-48a8-85d3-9248e775687a-kube-api-access-6d2rz\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:15.356258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.355342    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:15.356258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.356132    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:15.357159 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.357137    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:05:15.523524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.523072    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:15.523524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.523441    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:15.524421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.524401    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:05:15.697588 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.697563    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"
Oct 30 22:05:15.697764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.697592    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5" event={"ID":"bbe5eb62-a1f5-48a8-85d3-9248e775687a","Type":"ContainerDied","Data":"c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304"}
Oct 30 22:05:15.697764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:15.697649    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64718d916f9ecab86cda44032b2c33cb269d7affc88772fbc13ea1c2356f304"
Oct 30 22:05:23.608777 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:05:23.608742    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:05:26.086836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.086805    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:05:26.087346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.087326    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bbe5eb62-a1f5-48a8-85d3-9248e775687a" containerName="revoke-keys"
Oct 30 22:05:26.087416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.087350    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe5eb62-a1f5-48a8-85d3-9248e775687a" containerName="revoke-keys"
Oct 30 22:05:26.087452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.087436    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe5eb62-a1f5-48a8-85d3-9248e775687a" containerName="revoke-keys"
Oct 30 22:05:26.090741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.090726    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160632    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160751    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160791    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzsr\" (UniqueName: \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160827    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160853    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.160999 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.160952    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.195604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.195575    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"default-dockercfg-kg9lm\""
Oct 30 22:05:26.195790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.195770    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"kube-root-ca.crt\""
Oct 30 22:05:26.195836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.195786    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"distinct-w6-2182a5-kserve-self-signed-certs\""
Oct 30 22:05:26.195872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.195852    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"openshift-service-ca.crt\""
Oct 30 22:05:26.198297 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.198280    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267452    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267527    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267568    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzsr\" (UniqueName: \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267705    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267742    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.267825    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.268599    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.268989    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.273645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.269226    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.280743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.280711    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.282047 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.282021    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.332121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.332092    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzsr\" (UniqueName: \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\") pod \"distinct-w6-2182a5-kserve-97c95cdb6-mjhcm\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.402788 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.402697    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:26.785292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:26.785265    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:05:26.786702 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:05:26.786653    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35c8ac26_d502_4557_a7d6_535211e12e8b.slice/crio-b74ffadd05766253f5d5ffaee8c1932220cbc855b0934abdd2a936dd03e6c684 WatchSource:0}: Error finding container b74ffadd05766253f5d5ffaee8c1932220cbc855b0934abdd2a936dd03e6c684: Status 404 returned error can't find the container with id b74ffadd05766253f5d5ffaee8c1932220cbc855b0934abdd2a936dd03e6c684
Oct 30 22:05:27.757889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:27.757851    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" event={"ID":"35c8ac26-d502-4557-a7d6-535211e12e8b","Type":"ContainerStarted","Data":"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"}
Oct 30 22:05:27.758341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:27.757901    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" event={"ID":"35c8ac26-d502-4557-a7d6-535211e12e8b","Type":"ContainerStarted","Data":"b74ffadd05766253f5d5ffaee8c1932220cbc855b0934abdd2a936dd03e6c684"}
Oct 30 22:05:27.878085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:27.878025    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podStartSLOduration=1.878005447 podStartE2EDuration="1.878005447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:27.87783452 +0000 UTC m=+930.520428313" watchObservedRunningTime="2026-10-30 22:05:27.878005447 +0000 UTC m=+930.520599237"
Oct 30 22:05:28.403717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:28.403683    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:36.403219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:36.403184    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:36.415851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:36.415823    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:36.816588 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:36.816559    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:05:45.308186 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.308153    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:05:45.311961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.311940    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402275    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402306    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402324    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402460    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402481    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402497    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402518    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qlh\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402737    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.402823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.402801    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.418898 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.418871    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:05:45.503642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503525    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qlh\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.503642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503589    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.503642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503616    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503655    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503697    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503722    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503821    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503870    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.503894    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.504114    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.504181    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.504322    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.504338    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.504537    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.506230    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.508932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.506429    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.567217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.567151    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.567383 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.567359    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qlh\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:45.625571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:45.625544    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:46.053889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.053861    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:05:46.055442 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:05:46.055413    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb54e478_285c_480e_a2aa_7bd1f3df90d2.slice/crio-2099a2dd41fbcf000163e6a583f282b589d6a562fea3d01cc67bc52c83ba4c2e WatchSource:0}: Error finding container 2099a2dd41fbcf000163e6a583f282b589d6a562fea3d01cc67bc52c83ba4c2e: Status 404 returned error can't find the container with id 2099a2dd41fbcf000163e6a583f282b589d6a562fea3d01cc67bc52c83ba4c2e
Oct 30 22:05:46.057741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.057709    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.057815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.057772    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.057815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.057800    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.882073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.880832    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" event={"ID":"bb54e478-285c-480e-a2aa-7bd1f3df90d2","Type":"ContainerStarted","Data":"edf61fc359a0f2beae2311fc0a9149d8f4802ef74148e04751f680f1a053cba4"}
Oct 30 22:05:46.882073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:46.880898    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" event={"ID":"bb54e478-285c-480e-a2aa-7bd1f3df90d2","Type":"ContainerStarted","Data":"2099a2dd41fbcf000163e6a583f282b589d6a562fea3d01cc67bc52c83ba4c2e"}
Oct 30 22:05:47.010489 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:47.010413    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podStartSLOduration=2.010393061 podStartE2EDuration="2.010393061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:47.007241049 +0000 UTC m=+949.649834852" watchObservedRunningTime="2026-10-30 22:05:47.010393061 +0000 UTC m=+949.652986850"
Oct 30 22:05:47.628030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:47.627996    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:47.631690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:47.630397    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:05:47.631690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:47.630453    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 22:05:48.625967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:48.625935    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:05:48.626369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:48.625997    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 22:05:49.625935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:49.625903    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:05:49.626114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:49.625955    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 22:05:50.626513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:50.626475    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:05:50.626935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:50.626535    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 22:05:51.626411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:51.626379    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:05:51.626586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:51.626445    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 22:05:52.907298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:52.907264    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:52.907777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:52.907688    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:52.908547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:05:52.908525    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:05:53.608263 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:05:53.608221    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:06:00.141061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.141019    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:06:00.146739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.146717    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.201630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.201600    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6a0a09-openshift-default-dockercfg-h7ds4\""
Oct 30 22:06:00.227510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227476    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227519    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227620    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227772 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227737    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227784    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227866    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.227925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.227905    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rz6m\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.228049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.228033    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.228118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.228104    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.255262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.255225    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:06:00.329344 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329309    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329402    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329427    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329460    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329502    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329531    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329589    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329619    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rz6m\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.329800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329706    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.330000 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.329860    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.330401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.330379    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.330537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.330449    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.331273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.331230    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.331471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.331454    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.332894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.332868    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.334151 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.334128    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.432563 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.431174    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.432563 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.431829    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rz6m\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.459571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.459540    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:00.770771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.770701    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:06:00.775382 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.775360    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.829801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.829775    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3783c2-openshift-default-dockercfg-4c7q9\""
Oct 30 22:06:00.835752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.835577    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.835752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.835701    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.835986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.835808    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.835986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.835845    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.836093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.836013    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mzk\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.836093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.836052    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.836093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.836076    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.836319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.836157    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.836319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.836203    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.839728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.839701    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:06:00.915540 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.915510    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:06:00.916848 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:06:00.916824    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd25fa86_1783_4a60_a3a9_60c2e858412c.slice/crio-98369256faaf80a42c6c9f6c961e8ea125d0318b8859aa9b9f616786de2ac62d WatchSource:0}: Error finding container 98369256faaf80a42c6c9f6c961e8ea125d0318b8859aa9b9f616786de2ac62d: Status 404 returned error can't find the container with id 98369256faaf80a42c6c9f6c961e8ea125d0318b8859aa9b9f616786de2ac62d
Oct 30 22:06:00.919328 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.919295    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:00.919404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.919364    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:00.919404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.919396    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:00.936800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.936779    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.936895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.936814    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.936935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.936895    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mzk\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937015    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937083 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937062    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937114    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937149    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937242    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937317 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937291    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937375 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937317    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937498    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937537    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937572    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.937886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.937848    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.939217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.939198    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.939470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.939454    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:00.988576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.988544    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" event={"ID":"bd25fa86-1783-4a60-a3a9-60c2e858412c","Type":"ContainerStarted","Data":"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"}
Oct 30 22:06:00.988733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:00.988604    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" event={"ID":"bd25fa86-1783-4a60-a3a9-60c2e858412c","Type":"ContainerStarted","Data":"98369256faaf80a42c6c9f6c961e8ea125d0318b8859aa9b9f616786de2ac62d"}
Oct 30 22:06:01.000427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.000405    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:01.000869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.000849    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mzk\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:01.089332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.089251    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:01.115490 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.115412    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podStartSLOduration=1.115394183 podStartE2EDuration="1.115394183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:01.112324832 +0000 UTC m=+963.754918625" watchObservedRunningTime="2026-10-30 22:06:01.115394183 +0000 UTC m=+963.757987977"
Oct 30 22:06:01.460209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.460177    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:01.461888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.461853    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:01.462023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.461900    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:01.481938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.481912    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:06:01.482903 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:06:01.482875    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22165712_d558_45ac_ba97_32d060e970d3.slice/crio-d6fd738fd180bb4e81cc8d5e6a735e17fd9a928754360ce4b064b5bfb77f206b WatchSource:0}: Error finding container d6fd738fd180bb4e81cc8d5e6a735e17fd9a928754360ce4b064b5bfb77f206b: Status 404 returned error can't find the container with id d6fd738fd180bb4e81cc8d5e6a735e17fd9a928754360ce4b064b5bfb77f206b
Oct 30 22:06:01.485508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.485468    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:01.485605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.485548    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:01.485605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:01.485586    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:02.022178 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.022137    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" event={"ID":"22165712-d558-45ac-ba97-32d060e970d3","Type":"ContainerStarted","Data":"f2b0933be2487cfa468163a5d15e3f83698ae40caa7e3cb87c34004cf046bcb7"}
Oct 30 22:06:02.022377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.022204    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" event={"ID":"22165712-d558-45ac-ba97-32d060e970d3","Type":"ContainerStarted","Data":"d6fd738fd180bb4e81cc8d5e6a735e17fd9a928754360ce4b064b5bfb77f206b"}
Oct 30 22:06:02.097705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.092021    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:02.097705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.095785    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:02.097705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.095851    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:02.147308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.147245    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podStartSLOduration=2.14722819 podStartE2EDuration="2.14722819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:02.146402431 +0000 UTC m=+964.788996222" watchObservedRunningTime="2026-10-30 22:06:02.14722819 +0000 UTC m=+964.789821979"
Oct 30 22:06:02.460128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.460093    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:02.460343 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:02.460152    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:03.101076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:03.101040    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:03.101399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:03.101355    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:03.460497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:03.460465    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:03.460911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:03.460516    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:04.095689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:04.091967    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:04.095689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:04.092024    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:04.460133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:04.460104    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:04.460374 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:04.460350    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:05.107688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:05.102013    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:05.107688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:05.102073    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:05.460316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:05.460281    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:05.460496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:05.460329    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:06.111688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:06.111037    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:06.111688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:06.111100    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:06.460120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:06.460087    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:06:06.460302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:06.460139    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 22:06:07.094076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:07.094038    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:07.094359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:07.094329    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 22:06:07.693305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:07.693268    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:07.694235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:07.693958    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:07.694827 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:07.694801    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:06:08.319970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:08.319944    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:08.320444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:08.320415    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:08.321194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:08.321176    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:06:23.610869 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:06:23.610832    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:06:35.572687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.556190    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:06:35.572687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.561970    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.603889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.603719    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:06:35.668473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.668421    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"openshift-service-ca.crt\""
Oct 30 22:06:35.668473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.668426    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"kube-root-ca.crt\""
Oct 30 22:06:35.668473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.668489    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"distinct-w4-bce513-kserve-self-signed-certs\""
Oct 30 22:06:35.668878 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.668580    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"default-dockercfg-tjwnv\""
Oct 30 22:06:35.687834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.687795    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l6t\" (UniqueName: \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.687992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.687957    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.688081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.688054    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.688139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.688121    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.688197 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.688150    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.688258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.688221    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791415    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791571    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l6t\" (UniqueName: \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791693    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791735    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791786    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.791811    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.792271    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.792692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.792514    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.811686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.793774    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.815685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.813651    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.836816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.836707    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.877724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.873737    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l6t\" (UniqueName: \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\") pod \"distinct-w4-bce513-kserve-dcd445b56-4qrfp\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:35.910688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:35.909888    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:36.383138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:36.383110    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:06:36.383846 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:06:36.383818    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a15cfb_0ca8_4515_bb19_04400008e10f.slice/crio-2c5139fb096ac146c6f9b69a5345529313172845beed32c4ac261d05c2257119 WatchSource:0}: Error finding container 2c5139fb096ac146c6f9b69a5345529313172845beed32c4ac261d05c2257119: Status 404 returned error can't find the container with id 2c5139fb096ac146c6f9b69a5345529313172845beed32c4ac261d05c2257119
Oct 30 22:06:37.259419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:37.259382    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" event={"ID":"f5a15cfb-0ca8-4515-bb19-04400008e10f","Type":"ContainerStarted","Data":"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"}
Oct 30 22:06:37.259929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:37.259432    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" event={"ID":"f5a15cfb-0ca8-4515-bb19-04400008e10f","Type":"ContainerStarted","Data":"2c5139fb096ac146c6f9b69a5345529313172845beed32c4ac261d05c2257119"}
Oct 30 22:06:37.381872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:37.381811    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podStartSLOduration=2.381797372 podStartE2EDuration="2.381797372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:37.380539286 +0000 UTC m=+1000.023133077" watchObservedRunningTime="2026-10-30 22:06:37.381797372 +0000 UTC m=+1000.024391160"
Oct 30 22:06:37.911298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:37.911263    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:45.526508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.526471    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:06:45.530176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.530161    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.639300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.639269    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:06:45.654210 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654169    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654226    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654301    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654369    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654409    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654501 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654464    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654540 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654520    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654564    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.654640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.654623    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8xr\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.755464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.755404    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.755988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.755963    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.756181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.756163    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.756345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.756328    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.756509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.756488    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8xr\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.756687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.756647    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.757016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.756995    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.757260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.757222    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.757446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.757427    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.758300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.758278    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.777414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.766809    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.777414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.767080    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.777414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.767390    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.777414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.772729    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.778262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.778239    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.803647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.795885    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.820309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.820270    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:06:45.834438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.832821    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8xr\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.834438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.833853    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.838986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.838965    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.842830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.842809    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:45.881945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.881911    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:06:45.917718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.915771    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:45.930801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.930774    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:45.960430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960313    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960428    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960547    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960587    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960615    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960694    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tvb\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.960847 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960799    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.961223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.960988    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:45.961223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:45.961102    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.062922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.062838    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.062922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.062879    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.062922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.062898    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.062939    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tvb\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.063034    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.063075    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.063115    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.063154    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.063214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.063181    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.064319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.064280    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.064454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.064337    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.071846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.069853    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.071846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.069883    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.071846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.070052    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.071846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.070406    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.074922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.074069    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.126425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.126392    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.126769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.126739    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tvb\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.128754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.128729    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:06:46.133557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.133535    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.164767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.164731    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.164903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.164867    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.164983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.164912    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.164983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.164940    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj4g4\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.165128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.165107    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.165200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.165147    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.165200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.165175    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.165321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.165301    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.165456 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.165350    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.184433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.184410    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:46.195814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.195787    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:06:46.266937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.266904    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.266968    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267022    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267071    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267095    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267247    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj4g4\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267432    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267472    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267495    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267505    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267434    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267985 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267697    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.267985 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.267901    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.268103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.268021    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.270284 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.270258    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.270543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.270521    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.283341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.283302    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:06:46.287271 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:06:46.287244    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d64d0_1ec7_4416_a917_01735590a78c.slice/crio-b678fd72263f632637b076450a2519c9b615a1511e3e9a452ce888aac973c2cc WatchSource:0}: Error finding container b678fd72263f632637b076450a2519c9b615a1511e3e9a452ce888aac973c2cc: Status 404 returned error can't find the container with id b678fd72263f632637b076450a2519c9b615a1511e3e9a452ce888aac973c2cc
Oct 30 22:06:46.290445 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.290405    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.290554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.290498    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.290554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.290543    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.306719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.306689    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" event={"ID":"db4d64d0-1ec7-4416-a917-01735590a78c","Type":"ContainerStarted","Data":"b678fd72263f632637b076450a2519c9b615a1511e3e9a452ce888aac973c2cc"}
Oct 30 22:06:46.318870 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.318820    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:06:46.333503 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.333480    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.336377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.336357    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj4g4\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.446619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.446578    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:46.571888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.571844    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:06:46.578318 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:06:46.576950    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64060740_fbde_4041_ad6d_a8311b91c08b.slice/crio-0db3bbce71e4c1d7b47541d8c6534c102268fd13533ecc86e94df68925ce1cd8 WatchSource:0}: Error finding container 0db3bbce71e4c1d7b47541d8c6534c102268fd13533ecc86e94df68925ce1cd8: Status 404 returned error can't find the container with id 0db3bbce71e4c1d7b47541d8c6534c102268fd13533ecc86e94df68925ce1cd8
Oct 30 22:06:46.580216 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.580177    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.580327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.580256    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.580327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.580293    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.851909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.851830    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:06:46.858014 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.857964    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.858134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.858063    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.858134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:46.858105    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:47.349648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.349607    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" event={"ID":"b75548ab-6bc9-4ba9-aafe-66661f5c280d","Type":"ContainerStarted","Data":"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"}
Oct 30 22:06:47.349873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.349703    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" event={"ID":"b75548ab-6bc9-4ba9-aafe-66661f5c280d","Type":"ContainerStarted","Data":"6e5c94a2f589881be980f3fd501b0d427828eb56327f148f384f3a06cc51ccee"}
Oct 30 22:06:47.352473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.352438    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" event={"ID":"64060740-fbde-4041-ad6d-a8311b91c08b","Type":"ContainerStarted","Data":"3233b8cb4bdeff4df00030523886697ad7a9384a6da21eb2589f49f893c57ed6"}
Oct 30 22:06:47.352616 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.352500    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" event={"ID":"64060740-fbde-4041-ad6d-a8311b91c08b","Type":"ContainerStarted","Data":"0db3bbce71e4c1d7b47541d8c6534c102268fd13533ecc86e94df68925ce1cd8"}
Oct 30 22:06:47.356523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.356491    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" event={"ID":"db4d64d0-1ec7-4416-a917-01735590a78c","Type":"ContainerStarted","Data":"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"}
Oct 30 22:06:47.447688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.447071    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:47.468690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.461815    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.468690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.461881    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:47.476685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.475273    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podStartSLOduration=1.475257069 podStartE2EDuration="1.475257069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.474605603 +0000 UTC m=+1010.117199391" watchObservedRunningTime="2026-10-30 22:06:47.475257069 +0000 UTC m=+1010.117850859"
Oct 30 22:06:47.614750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.614143    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podStartSLOduration=2.614122371 podStartE2EDuration="2.614122371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.611759616 +0000 UTC m=+1010.254353403" watchObservedRunningTime="2026-10-30 22:06:47.614122371 +0000 UTC m=+1010.256716161"
Oct 30 22:06:47.746109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.746046    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podStartSLOduration=2.746025682 podStartE2EDuration="2.746025682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.742414654 +0000 UTC m=+1010.385008447" watchObservedRunningTime="2026-10-30 22:06:47.746025682 +0000 UTC m=+1010.388619474"
Oct 30 22:06:47.845059 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.845029    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:47.848843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.848812    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.849084 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:47.849057    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:48.184771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.184737    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:48.186215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.186189    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.186321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.186236    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:48.448747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.447210    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.448747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.447265    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:48.845748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.845014    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.845748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:48.845074    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:49.185457 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.185426    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.185606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.185482    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:49.451535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.451444    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.451809 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.451778    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:49.847745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.847074    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.847745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:49.847139    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:50.184945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.184910    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.185105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.184959    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:50.448743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.447904    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.448743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.447962    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:50.844171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.844070    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.844454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:50.844422    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:51.185508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.185474    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.185920 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.185523    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:51.447688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.447121    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.447688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.447188    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:51.843745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.843178    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.843745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:51.843238    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:52.185699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.185654    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.186113 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.185718    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:52.448744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.447736    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.448744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.447799    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:52.845611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.845057    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.845611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:52.845118    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 22:06:53.184936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.184897    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:06:53.185131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.184983    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 22:06:53.447869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.447768    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:06:53.447869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.447844    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 22:06:53.622721 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:06:53.622649    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:06:53.851141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.851050    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:53.851583 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.851563    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:53.852361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:53.852331    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:06:54.197202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.197169    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:54.199303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.199268    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:54.200198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.200180    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:06:54.513704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.513565    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:54.514338 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.514076    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:06:54.516206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:06:54.516185    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:07:23.639858 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:07:23.639775    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:07:33.630483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.630448    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:07:33.635775 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.635758    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716288 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716254    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n9f\" (UniqueName: \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716295    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716312    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716342    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716477    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.716574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.716499    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.738965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.738936    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:07:33.741093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.741079    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"distinct-2-w6-2182a5-kserve-self-signed-certs\""
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.819533    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.819622    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.819652    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.819973    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n9f\" (UniqueName: \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.820040    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.820593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.820062    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.824231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.824189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.830767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.824647    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.830767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.825926    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.830767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.829333    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.831932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.831906    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.889624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.889550    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n9f\" (UniqueName: \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\") pod \"distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") " pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:33.951044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:33.951016    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:34.444064 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:34.444040    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:07:34.445265 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:07:34.445238    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod405f22c9_78f4_4763_95d0_444646c01cf4.slice/crio-c627133a0581cc6aeb1765391054974bd420828fee61fccf496734d6bc03d193 WatchSource:0}: Error finding container c627133a0581cc6aeb1765391054974bd420828fee61fccf496734d6bc03d193: Status 404 returned error can't find the container with id c627133a0581cc6aeb1765391054974bd420828fee61fccf496734d6bc03d193
Oct 30 22:07:34.662897 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:34.662852    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" event={"ID":"405f22c9-78f4-4763-95d0-444646c01cf4","Type":"ContainerStarted","Data":"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"}
Oct 30 22:07:34.662897 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:34.662892    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" event={"ID":"405f22c9-78f4-4763-95d0-444646c01cf4","Type":"ContainerStarted","Data":"c627133a0581cc6aeb1765391054974bd420828fee61fccf496734d6bc03d193"}
Oct 30 22:07:34.787223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:34.787125    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podStartSLOduration=1.787111557 podStartE2EDuration="1.787111557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:34.786612749 +0000 UTC m=+1057.429206529" watchObservedRunningTime="2026-10-30 22:07:34.787111557 +0000 UTC m=+1057.429705347"
Oct 30 22:07:34.953349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:34.951879    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:43.951519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:43.951482    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:43.964698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:43.964645    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:44.734893 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:07:44.734861    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:07:53.608983 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:07:53.608944    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:08:02.715497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.715462    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:08:02.719396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.719377    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.827901 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.827871    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:08:02.895905 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.895877    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.895926    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896014    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsv5t\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896047    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896063    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896151    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896175    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896230    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.896387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.896355    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997174    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997219    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997288    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997343    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997436    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997484    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997533    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsv5t\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997585    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997608    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997749    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.997856 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.997831    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.998105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.998082    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.998170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.998139    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:02.998219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:02.998175    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.000044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.000019    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.000259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.000242    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.061892 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.061861    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.062045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.062015    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsv5t\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.333056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.332969    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:03.815093 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:08:03.815062    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f4016c_b847_4ee1_a28c_b819f8ce95af.slice/crio-d576ee8b018b183887f7d265df6c5f18cd9cad348645a9d653bb4cb4063af0b8 WatchSource:0}: Error finding container d576ee8b018b183887f7d265df6c5f18cd9cad348645a9d653bb4cb4063af0b8: Status 404 returned error can't find the container with id d576ee8b018b183887f7d265df6c5f18cd9cad348645a9d653bb4cb4063af0b8
Oct 30 22:08:03.815856 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.815832    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:08:03.818700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.818627    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.818799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.818753    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.818868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.818800    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.828476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:03.828448    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" event={"ID":"d0f4016c-b847-4ee1-a28c-b819f8ce95af","Type":"ContainerStarted","Data":"d576ee8b018b183887f7d265df6c5f18cd9cad348645a9d653bb4cb4063af0b8"}
Oct 30 22:08:04.834996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:04.834960    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" event={"ID":"d0f4016c-b847-4ee1-a28c-b819f8ce95af","Type":"ContainerStarted","Data":"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"}
Oct 30 22:08:05.081176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:05.081117    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podStartSLOduration=3.081102246 podStartE2EDuration="3.081102246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:05.029694516 +0000 UTC m=+1087.672288302" watchObservedRunningTime="2026-10-30 22:08:05.081102246 +0000 UTC m=+1087.723696097"
Oct 30 22:08:05.333301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:05.333272    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:05.335056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:05.335031    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:05.335194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:05.335084    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:06.333976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:06.333943    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:06.334362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:06.334016    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:07.334223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:07.334188    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:07.334589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:07.334244    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:08.333504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:08.333470    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:08.333705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:08.333528    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:09.333344 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:09.333301    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:09.333913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:09.333361    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:10.333902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:10.333864    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:10.334332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:10.333934    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 22:08:11.338036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:11.338001    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:11.338487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:11.338312    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:11.339206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:11.339186    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:08:23.610247 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:08:23.610198    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:08:31.387989 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.387958    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:08:31.391928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.391911    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.438469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.438435    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:08:31.442286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.442266    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.445007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.444975    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445046    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445087    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445233 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445170    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445233 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445192    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445306    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7fh\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445413    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445495 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445447    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.445550 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.445530    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.509316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.508641    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:08:31.528546 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.528503    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:08:31.546740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.546692    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.546936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.546752    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.546936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.546787    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.546936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.546835    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.546950    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547003    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547042    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm7l\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547080    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547114    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547139    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547173    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547275    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7fh\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547329    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547390    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547389    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547475    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547508    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547528    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547882 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547622    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.547882 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547651    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.547882 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547716    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.548038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.547946    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.548630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.548602    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.549588 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.549566    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.550895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.550875    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.613317 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.613285    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.613516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.613494    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7fh\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.648597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648512    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm7l\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.648597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648565    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.648597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648593    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.648900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648813    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.648900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648867    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.648988    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649010    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649034    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649076    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649085    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649115    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649247    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649356    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.649593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.649570    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.651103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.651078    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.651484 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.651467    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.706520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.706483    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:31.711959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.711931    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.712240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.712207    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm7l\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:31.756198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:31.756158    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:32.138410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.138384    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:08:32.139516 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:08:32.139488    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46d198d_51f4_4a97_8135_74d9b02ebf2c.slice/crio-1aabf3f3e6eca58357fb3247b70ea5f140e1885a1e281f7e9309dd4656057a2f WatchSource:0}: Error finding container 1aabf3f3e6eca58357fb3247b70ea5f140e1885a1e281f7e9309dd4656057a2f: Status 404 returned error can't find the container with id 1aabf3f3e6eca58357fb3247b70ea5f140e1885a1e281f7e9309dd4656057a2f
Oct 30 22:08:32.141955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.141911    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.142042 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.141988    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.142081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.142024    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.195556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.195535    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:08:32.197847 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:08:32.197816    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6568c8_6bbe_49d8_8acc_c267104ae995.slice/crio-4403f44878acf0a90eb80f3f35c545377dd8bb1a4104a398d7990674aa8fa527 WatchSource:0}: Error finding container 4403f44878acf0a90eb80f3f35c545377dd8bb1a4104a398d7990674aa8fa527: Status 404 returned error can't find the container with id 4403f44878acf0a90eb80f3f35c545377dd8bb1a4104a398d7990674aa8fa527
Oct 30 22:08:32.200567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.200538    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.200638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.200605    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.200692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:32.200635    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:33.041360 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.041317    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" event={"ID":"7d6568c8-6bbe-49d8-8acc-c267104ae995","Type":"ContainerStarted","Data":"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"}
Oct 30 22:08:33.041857 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.041411    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" event={"ID":"7d6568c8-6bbe-49d8-8acc-c267104ae995","Type":"ContainerStarted","Data":"4403f44878acf0a90eb80f3f35c545377dd8bb1a4104a398d7990674aa8fa527"}
Oct 30 22:08:33.042952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.042925    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" event={"ID":"b46d198d-51f4-4a97-8135-74d9b02ebf2c","Type":"ContainerStarted","Data":"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"}
Oct 30 22:08:33.043044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.042979    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" event={"ID":"b46d198d-51f4-4a97-8135-74d9b02ebf2c","Type":"ContainerStarted","Data":"1aabf3f3e6eca58357fb3247b70ea5f140e1885a1e281f7e9309dd4656057a2f"}
Oct 30 22:08:33.227849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.227792    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podStartSLOduration=2.227777724 podStartE2EDuration="2.227777724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.227466736 +0000 UTC m=+1115.870060553" watchObservedRunningTime="2026-10-30 22:08:33.227777724 +0000 UTC m=+1115.870371513"
Oct 30 22:08:33.356962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.356866    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podStartSLOduration=2.356850919 podStartE2EDuration="2.356850919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.355011767 +0000 UTC m=+1115.997605569" watchObservedRunningTime="2026-10-30 22:08:33.356850919 +0000 UTC m=+1115.999444708"
Oct 30 22:08:33.730691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.710443    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:33.730691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.710611    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:33.730691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.710689    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:33.757235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.757196    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:33.758632 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.758610    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:33.758772 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:33.758651    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:34.712690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:34.709047    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:34.712690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:34.709123    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:34.756754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:34.756717    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:34.756918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:34.756772    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:35.711689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:35.707979    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:35.711689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:35.708053    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:35.756919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:35.756884    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:35.757285 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:35.756931    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:36.707690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:36.707076    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:36.707690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:36.707141    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:36.757563 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:36.757523    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:36.757975 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:36.757575    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:37.717096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:37.717054    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:37.717403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:37.717371    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:37.757188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:37.757148    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:37.757420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:37.757396    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:38.707109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:38.707068    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:08:38.707742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:38.707711    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 22:08:38.756730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:38.756654    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:08:38.756906 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:38.756764    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 22:08:39.796002 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:39.795957    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:39.797245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:39.797228    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:39.798906 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:39.798885    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:08:40.155473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:40.155362    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:40.155911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:40.155891    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:40.156697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:40.156647    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:08:45.666283 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.666252    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:08:45.670027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.670011    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.773739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.773712    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"distinct-2-w5-3783c2-kserve-self-signed-certs\""
Oct 30 22:08:45.776652 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.776631    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:08:45.787868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.787846    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.787984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.787878    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.788028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.787993    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.788072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.788032    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmxx\" (UniqueName: \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.788165 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.788149    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.788234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.788195    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.890118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.889540    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.890441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.890409    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmxx\" (UniqueName: \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.891074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.891051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.891296 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.891278    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.892309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.892031    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.892420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.892295    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.892478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.892460    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.892536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.892495    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.893086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.893062    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.895435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.895415    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.896122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.896106    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.954730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.954703    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmxx\" (UniqueName: \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\") pod \"distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:45.982504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:45.982474    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:46.367972 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:46.367946    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:08:46.368938 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:08:46.368915    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ec1674_4204_4c78_8c0e_fc4882173d37.slice/crio-e2fe5bbe7885047ecfa095f518f87119f442c1407c03f6a9d4abbad83e7e8a80 WatchSource:0}: Error finding container e2fe5bbe7885047ecfa095f518f87119f442c1407c03f6a9d4abbad83e7e8a80: Status 404 returned error can't find the container with id e2fe5bbe7885047ecfa095f518f87119f442c1407c03f6a9d4abbad83e7e8a80
Oct 30 22:08:47.184842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:47.184810    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" event={"ID":"05ec1674-4204-4c78-8c0e-fc4882173d37","Type":"ContainerStarted","Data":"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"}
Oct 30 22:08:47.184842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:47.184850    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" event={"ID":"05ec1674-4204-4c78-8c0e-fc4882173d37","Type":"ContainerStarted","Data":"e2fe5bbe7885047ecfa095f518f87119f442c1407c03f6a9d4abbad83e7e8a80"}
Oct 30 22:08:47.309894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:47.309823    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podStartSLOduration=2.30980795 podStartE2EDuration="2.30980795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:47.308569625 +0000 UTC m=+1129.951163415" watchObservedRunningTime="2026-10-30 22:08:47.30980795 +0000 UTC m=+1129.952401770"
Oct 30 22:08:47.982770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:47.982733    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:53.610834 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:08:53.610795    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:08:55.983578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:55.983542    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:55.996204 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:55.996178    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:08:56.245488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:08:56.245415    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:09:17.363958 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.363925    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:09:17.368138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.368112    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.463692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.463042    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:09:17.521567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521535    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521582    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521604    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521748    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521902    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521923    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.521955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521938    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppcd\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.522036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521957    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.522036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.521972    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623434    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623468    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623486    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppcd\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623504    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623524    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623612    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623679    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623719    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.623880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623784    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.624103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623956    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.624103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.623998    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.624103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.624041    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.624225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.624189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.624525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.624504    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.626227 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.626204    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.626403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.626386    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.686483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.686463    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.686579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.686534    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppcd\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:17.980915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:17.980875    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:18.306524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.306443    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:09:18.311826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.311794    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.374020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.373984    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:09:18.439018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.438972    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439228 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439081    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439228 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439112    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439228 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439192    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439286    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439403    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439482    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439537    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.439624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.439604    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdz4\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.464437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.464414    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:09:18.465471 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:18.465446    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f467322_1c70_4132_9a74_d755b3a1edb7.slice/crio-c74ee16b5bb690e7c166ea8c1074b3f9fff4faa0a0b604320bcfb3c7c4da911c WatchSource:0}: Error finding container c74ee16b5bb690e7c166ea8c1074b3f9fff4faa0a0b604320bcfb3c7c4da911c: Status 404 returned error can't find the container with id c74ee16b5bb690e7c166ea8c1074b3f9fff4faa0a0b604320bcfb3c7c4da911c
Oct 30 22:09:18.468046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.468011    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.468133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.468092    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.468178 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.468135    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.541210 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541182    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541239    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541278    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541304    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdz4\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541416    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541463    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541489    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541554    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541589    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541825 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541785    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541884 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541840    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.541982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.541958    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.542231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.542211    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.543050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.543025    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.543592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.543574    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.544200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.544182    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.605034 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.604955    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.605181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.605153    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdz4\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:18.626000 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:18.625965    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:19.061877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.061853    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:09:19.062930 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:19.062901    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d60e1_9571_4ea7_a2c9_4558e71203a8.slice/crio-14638b7c5fcf336e75eee04a2e2d49a8cebdfda295b8cf576e709b2ed00997c1 WatchSource:0}: Error finding container 14638b7c5fcf336e75eee04a2e2d49a8cebdfda295b8cf576e709b2ed00997c1: Status 404 returned error can't find the container with id 14638b7c5fcf336e75eee04a2e2d49a8cebdfda295b8cf576e709b2ed00997c1
Oct 30 22:09:19.065413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.065379    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.065485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.065444    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.065485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.065473    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.385535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.385436    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" event={"ID":"0f467322-1c70-4132-9a74-d755b3a1edb7","Type":"ContainerStarted","Data":"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"}
Oct 30 22:09:19.385535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.385498    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" event={"ID":"0f467322-1c70-4132-9a74-d755b3a1edb7","Type":"ContainerStarted","Data":"c74ee16b5bb690e7c166ea8c1074b3f9fff4faa0a0b604320bcfb3c7c4da911c"}
Oct 30 22:09:19.387269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.387244    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" event={"ID":"e48d60e1-9571-4ea7-a2c9-4558e71203a8","Type":"ContainerStarted","Data":"6ced5b311968ee967cc82423b51547789b005d15ec33696fb02912365d84d496"}
Oct 30 22:09:19.387393 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.387285    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" event={"ID":"e48d60e1-9571-4ea7-a2c9-4558e71203a8","Type":"ContainerStarted","Data":"14638b7c5fcf336e75eee04a2e2d49a8cebdfda295b8cf576e709b2ed00997c1"}
Oct 30 22:09:19.523691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.519831    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podStartSLOduration=2.519809295 podStartE2EDuration="2.519809295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.518928712 +0000 UTC m=+1162.161522513" watchObservedRunningTime="2026-10-30 22:09:19.519809295 +0000 UTC m=+1162.162403085"
Oct 30 22:09:19.626528 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.626501    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:19.647099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.647002    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podStartSLOduration=1.64698725 podStartE2EDuration="1.64698725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.645819325 +0000 UTC m=+1162.288413115" watchObservedRunningTime="2026-10-30 22:09:19.64698725 +0000 UTC m=+1162.289581038"
Oct 30 22:09:19.757045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.757013    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:09:19.760964 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.760948    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.824045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.824015    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"distinct-2-w3-8ca52a-kserve-self-signed-certs\""
Oct 30 22:09:19.832512 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.831616    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:09:19.862173 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862140    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.862349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862186    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.862349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862332    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.862465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862362    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wnw\" (UniqueName: \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.862557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862537    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.862597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.862578    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.966825    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.966975    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.967008    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wnw\" (UniqueName: \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.967222    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.967274    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.967685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.967346    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.969744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.969048    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.982984    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.983057    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.983375    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.983625    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.983911    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.989312    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:19.994120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:19.989443    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:20.060302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.043706    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wnw\" (UniqueName: \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\") pod \"distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:20.074739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.073897    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:20.645038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.644998    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:20.645621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.645590    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:20.740470 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:20.740421    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f0b6b0d_bcbc_480d_9bcb_8923372d8035.slice/crio-cb38babd51143e86ba080c1aa10ee4fb4fb459fb77c2bfcb48d3ab8ec9caa72a WatchSource:0}: Error finding container cb38babd51143e86ba080c1aa10ee4fb4fb459fb77c2bfcb48d3ab8ec9caa72a: Status 404 returned error can't find the container with id cb38babd51143e86ba080c1aa10ee4fb4fb459fb77c2bfcb48d3ab8ec9caa72a
Oct 30 22:09:20.742583 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.742470    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:09:20.981248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.981212    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:20.981403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:20.981273    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:21.547197 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.547153    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" event={"ID":"1f0b6b0d-bcbc-480d-9bcb-8923372d8035","Type":"ContainerStarted","Data":"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"}
Oct 30 22:09:21.547376 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.547212    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" event={"ID":"1f0b6b0d-bcbc-480d-9bcb-8923372d8035","Type":"ContainerStarted","Data":"cb38babd51143e86ba080c1aa10ee4fb4fb459fb77c2bfcb48d3ab8ec9caa72a"}
Oct 30 22:09:21.639144 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.639106    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:21.639358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.639168    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:21.679446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.679393    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podStartSLOduration=2.679378464 podStartE2EDuration="2.679378464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:21.678282812 +0000 UTC m=+1164.320876602" watchObservedRunningTime="2026-10-30 22:09:21.679378464 +0000 UTC m=+1164.321972296"
Oct 30 22:09:21.981688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.981640    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:21.981868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:21.981704    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:22.075324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:22.075289    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:22.638219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:22.638178    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:22.638432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:22.638227    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:22.982152 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:22.982099    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:22.982731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:22.982177    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:23.633308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:23.631038    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:23.633308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:23.631116    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:23.659908 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:09:23.659863    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:09:23.981961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:23.981925    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:23.982128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:23.981976    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:24.634688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:24.632055    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:24.634688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:24.632150    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:24.982060 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:24.982019    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:09:24.982237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:24.982102    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 22:09:25.627092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:25.627063    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:09:25.627265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:25.627126    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 22:09:26.292134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:26.290838    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:26.292134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:26.292071    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:26.293425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:26.293400    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:09:27.078682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:27.078631    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:27.080145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:27.079619    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:27.080877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:27.080848    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:09:30.076771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:30.076738    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:30.728792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:30.728743    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.812609 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:30.812568    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.813273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:30.812893    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.875547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:30.875492    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:09:31.052207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.052098    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.094689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.088904    4295 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 22:09:31.179792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.179745    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.236791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.236747    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.322770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.310747    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.479688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.477739    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.813551 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.813506    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.814213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.814193    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:09:31.902836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:31.902805    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:09:40.086063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:40.086031    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:40.097349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:40.097322    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:47.329654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.329615    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:09:47.333625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.333599    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.414372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.414334    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:09:47.417222 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417192    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwh2\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417264    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417422 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417379    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417479 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417464    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417559 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417543    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417592    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417633    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417681    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.417802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.417753    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519310    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519345    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519398    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519476    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwh2\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519521    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519558    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519627    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.519953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519933    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519952    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519928    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519995    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.520014    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.519999    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.520541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.520522    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.522144 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.522120    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.522241 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.522123    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.586087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.586032    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwh2\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.589024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.589000    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:47.647375 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:47.647355    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:48.081796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.081774    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:09:48.082891 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:48.082868    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eb4f10_5947_4a49_832f_4eaf57e3f7dc.slice/crio-da1399ed1acd36b2e685d3729c2ad6333d9e15fb9d47c490e00eabed2ece37b3 WatchSource:0}: Error finding container da1399ed1acd36b2e685d3729c2ad6333d9e15fb9d47c490e00eabed2ece37b3: Status 404 returned error can't find the container with id da1399ed1acd36b2e685d3729c2ad6333d9e15fb9d47c490e00eabed2ece37b3
Oct 30 22:09:48.085200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.085166    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.085296 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.085235    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.085296 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.085264    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.339985 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.339903    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:09:48.344073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.344051    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.364746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.364718    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:09:48.369383 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.369363    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.414688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.410655    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:09:48.434343 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434306    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434518 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434422    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269lj\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434518 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434459    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434518 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434484    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434531    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434554    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434597    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434626    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434675    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434723    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434815    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434842    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434868    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9cp\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434914    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434950    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.434987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.434983    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.435351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.435014    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.435351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.435042    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.449147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.449113    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:09:48.536034 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.535998    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536041    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536095    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536122    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536169    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536452    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536549 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536502    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536549 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536543    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536541    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536736    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536774    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536799    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9cp\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.536883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536851    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536930 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536900    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.536964 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536941    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.537001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.536977    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.537031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.537012    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.537151 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.537132    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.537307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.537291    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269lj\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.538098 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538068    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.538197 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538145    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.538785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538448    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.538785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538476    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.538785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538536    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.538785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538727    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.538785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538767    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.539071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.538796    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.539274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.539253    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.539398 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.539279    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.540442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.540420    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.541246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.541222    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.541419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.541400    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.601293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.601186    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.601769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.601743    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.601850 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.601743    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269lj\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.601915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.601885    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9cp\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.660650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.660611    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:48.683002 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.682979    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:48.841506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.841122    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:09:48.851823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.851723    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" event={"ID":"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc","Type":"ContainerStarted","Data":"cb3cd638dedff9f8f2d9161b7e1ec21c6a73f9d5ea13ef1cb29411b684c68de4"}
Oct 30 22:09:48.851823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.851755    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" event={"ID":"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc","Type":"ContainerStarted","Data":"da1399ed1acd36b2e685d3729c2ad6333d9e15fb9d47c490e00eabed2ece37b3"}
Oct 30 22:09:48.851956 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.851835    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.911052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.911021    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9cff60-openshift-default-dockercfg-z696x\""
Oct 30 22:09:48.928037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.928001    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:09:48.946414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946374    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.946574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946470    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.946574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946515    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.946789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946762    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.946930 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946907    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.947014 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.946996    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt889\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.947177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.947157    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.947254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.947217    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.947313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.947270    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:48.985650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:48.985595    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podStartSLOduration=1.985582247 podStartE2EDuration="1.985582247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:48.982176731 +0000 UTC m=+1191.624770521" watchObservedRunningTime="2026-10-30 22:09:48.985582247 +0000 UTC m=+1191.628176038"
Oct 30 22:09:49.048389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048357    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt889\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048420    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048461    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048497    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048580    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048613    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048703    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048855    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.048969 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048918    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.049020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048991    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.049020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.048996    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.049306 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.049289    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.049421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.049396    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.049620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.049601    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.050962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.050941    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.051130 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.051111    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.073815 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:49.073792    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff28f3af_74a4_42a5_997d_6efe9365edb3.slice/crio-f9d41773976eccf42d5db1b519cd21d04a2a7d2c06d8973eb6d5e0328ba0f57f WatchSource:0}: Error finding container f9d41773976eccf42d5db1b519cd21d04a2a7d2c06d8973eb6d5e0328ba0f57f: Status 404 returned error can't find the container with id f9d41773976eccf42d5db1b519cd21d04a2a7d2c06d8973eb6d5e0328ba0f57f
Oct 30 22:09:49.074227 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.074206    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:09:49.076565 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.076532    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.076653 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.076593    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.076653 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.076621    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.116519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.116441    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt889\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.116936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.116906    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.154547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.154522    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:09:49.155429 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:09:49.155409    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcf0aab_a38d_4e10_b8b4_a140f4ecaba5.slice/crio-6f634e3a0c0579e35308a5513b70bd3b793df184fdc1830b2db488440a7caf04 WatchSource:0}: Error finding container 6f634e3a0c0579e35308a5513b70bd3b793df184fdc1830b2db488440a7caf04: Status 404 returned error can't find the container with id 6f634e3a0c0579e35308a5513b70bd3b793df184fdc1830b2db488440a7caf04
Oct 30 22:09:49.158435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.158402    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.158504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.158462    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.158504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.158497    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.170053 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.170031    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:49.626558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.626523    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:09:49.635971 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.635924    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.636072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.636019    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.636148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.636067    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.649101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.649036    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:49.650611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.650398    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:49.650611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.650448    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:49.891936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.888145    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" event={"ID":"ff28f3af-74a4-42a5-997d-6efe9365edb3","Type":"ContainerStarted","Data":"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"}
Oct 30 22:09:49.891936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.888207    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" event={"ID":"ff28f3af-74a4-42a5-997d-6efe9365edb3","Type":"ContainerStarted","Data":"f9d41773976eccf42d5db1b519cd21d04a2a7d2c06d8973eb6d5e0328ba0f57f"}
Oct 30 22:09:49.956412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.956370    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" event={"ID":"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5","Type":"ContainerStarted","Data":"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"}
Oct 30 22:09:49.960767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.960718    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" event={"ID":"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5","Type":"ContainerStarted","Data":"6f634e3a0c0579e35308a5513b70bd3b793df184fdc1830b2db488440a7caf04"}
Oct 30 22:09:49.973986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.972571    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" event={"ID":"006bff34-f902-46d0-8469-543ae5eced45","Type":"ContainerStarted","Data":"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"}
Oct 30 22:09:49.974253 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:49.974228    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" event={"ID":"006bff34-f902-46d0-8469-543ae5eced45","Type":"ContainerStarted","Data":"be6385b9e261be96e3c2772541d76650fd7bbd14f60e86b92c57df8863d0425b"}
Oct 30 22:09:50.081848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.077561    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podStartSLOduration=2.077538949 podStartE2EDuration="2.077538949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.053141997 +0000 UTC m=+1192.695735780" watchObservedRunningTime="2026-10-30 22:09:50.077538949 +0000 UTC m=+1192.720132739"
Oct 30 22:09:50.179746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.179702    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:50.179924 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.179843    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.179924 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.179888    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:50.222691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.222430    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podStartSLOduration=2.222409548 podStartE2EDuration="2.222409548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.19612478 +0000 UTC m=+1192.838718571" watchObservedRunningTime="2026-10-30 22:09:50.222409548 +0000 UTC m=+1192.865003338"
Oct 30 22:09:50.654272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.654170    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.654272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.654239    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:50.666464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.666423    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:50.668796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.668764    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.669004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.668978    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:50.685825 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.685793    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:50.686912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.686886    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.687083 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:50.687060    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:51.174694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.174051    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.174694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.174114    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:51.672689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.652141    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.672689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.652208    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:51.699693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.698804    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.699693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.698869    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:51.706688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.701797    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.706688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.701842    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:51.744726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.738023    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:51.832688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.820757    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:51.908703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:51.907817    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.037689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.029275    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.091690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.090881    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.171688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.169739    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.175686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.172764    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.175686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.172819    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:52.253689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.236761    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:52.321744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.321152    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.474692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.472786    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.600135 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.600026    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:09:52.648146 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.648111    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.648463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.648417    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:52.662160 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.662128    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.662432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.662394    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:52.687085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.687040    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.687644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.687594    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:52.839729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.839680    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:52.840384 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.840355    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:09:52.923737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.909320    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:52.932927 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.931793    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:09:53.004688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:52.997452    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podStartSLOduration=4.997432644 podStartE2EDuration="4.997432644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.384932329 +0000 UTC m=+1193.027526122" watchObservedRunningTime="2026-10-30 22:09:52.997432644 +0000 UTC m=+1195.640026434"
Oct 30 22:09:53.093861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.093815    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:09:53.122575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.122538    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:53.137689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.124644    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:09:53.138516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.138493    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:09:53.188818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.183754    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:53.209875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.209823    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:09:53.229145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.228950    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.229145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.229016    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:53.293688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.261423    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:53.293688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.269059    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:09:53.334689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.324762    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:53.334689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.325236    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:09:53.674608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.659093    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.674608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.659170    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:53.698724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.688991    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.698724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.689051    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:53.698724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.693722    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.698724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.693796    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:53.841743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:53.841695    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.077697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.061493    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.267890 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.267684    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.275995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.275960    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.350741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.274655    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.355685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.350900    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.367012 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:09:54.366962    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:09:54.393814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.367303    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:54.394118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.367311    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.430547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.397516    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.526687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.524217    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.719435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.719402    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.720043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.720010    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:54.725439 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.725402    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.732810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.732781    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.780722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.759053    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.780722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.759103    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:54.780722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.759164    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.781351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.781320    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.781537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:54.781512    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:55.060992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.060894    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.128996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.128947    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.236724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.236691    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.236984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.236957    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:55.291691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.290796    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.363104 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.362978    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:09:55.396937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.396894    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.397240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.396825    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.438032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.437991    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.502807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.502759    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.524864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.524826    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.549937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.549905    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:09:55.661520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.661430    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.661806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.661777    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:55.662022 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.662004    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.662154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.662132    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 22:09:55.763691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.727774    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.763691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.727833    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:55.763691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.727903    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.763691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.728279    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:09:55.806769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.784828    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:55.806769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:55.785360    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:09:56.084826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.084757    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:56.183081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.183050    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:09:56.248341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.248306    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.248610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.248585    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:56.351748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.302328    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:09:56.364712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.352034    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Oct 30 22:09:56.412310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.412271    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:09:56.486722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.453794    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:56.486722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.455952    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:56.646689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.646563    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:09:56.661871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.652254    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:09:56.670923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.670901    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:09:56.674174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.672851    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.674174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.672917    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 22:09:56.678918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.678901    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:09:56.680052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.680036    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:09:56.686693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.682378    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:09:56.686693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.685255    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:09:56.686693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.686002    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.686693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.686045    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:09:56.695686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.692095    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:09:56.781686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.778774    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:09:56.786771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.786747    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:09:56.967867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.967836    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:56.972579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.972555    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:56.974735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:56.974717    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:09:57.171048 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:57.171008    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:57.171232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:57.171059    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:58.153690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.145063    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:58.157529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.157493    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:58.166765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.166734    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:09:58.170461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.170432    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:09:58.171486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.171448    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 22:09:58.246841 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.246807    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:58.247834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.247797    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:58.249321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:58.249298    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:09:59.175440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:59.175411    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:59.175957 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:59.175771    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:09:59.176356 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:09:59.176339    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:10:06.828033 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:06.827984    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:06.847790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:06.847738    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:06.989800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:06.988507    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.299191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.299133    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.412700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.410754    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.412700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.411997    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.412700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.412349    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 22:10:07.462688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.462523    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.507105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.507062    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.672845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.672753    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.696271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.696236    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.746285 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:07.746245    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.438122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.438076    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.439633 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.439609    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:10:08.529814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.529754    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.546699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.543430    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:10:08.546699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.544971    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:10:08.586246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:08.586218    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:10:16.060254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:16.060174    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"]
Oct 30 22:10:16.111896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:16.111865    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d72420-9lkj5"]
Oct 30 22:10:17.496397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:17.496363    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe5eb62-a1f5-48a8-85d3-9248e775687a" path="/var/lib/kubelet/pods/bbe5eb62-a1f5-48a8-85d3-9248e775687a/volumes"
Oct 30 22:10:23.419273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:23.419239    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:10:23.419716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:23.419445    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" containerID="cri-o://f463d3e326b0a52bc7eb9ed8f89135eda437d078ef37e18f188c069af619eb7d" gracePeriod=30
Oct 30 22:10:23.610816 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:10:23.610778    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:10:28.400825 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.400773    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.541686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.531844    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.562637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.561800    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.591855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.591799    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.691868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.690215    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.691868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.690633    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:10:28.753691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.730386    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.781935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.781869    4295 generic.go:356] "Generic (PLEG): container finished" podID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerID="f463d3e326b0a52bc7eb9ed8f89135eda437d078ef37e18f188c069af619eb7d" exitCode=0
Oct 30 22:10:28.782361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.782281    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" event={"ID":"472d6bf4-6c02-44ca-97de-27d02cdd1035","Type":"ContainerDied","Data":"f463d3e326b0a52bc7eb9ed8f89135eda437d078ef37e18f188c069af619eb7d"}
Oct 30 22:10:28.801092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.801050    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:10:28.812785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.812743    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:28.850716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:28.848764    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.190686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.187535    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.299806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.299751    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.548745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.467053    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.548745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.467750    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:10:29.548745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.538835    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.580699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.563767    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.598880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.598820    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.731992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.711796    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.731992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.728702    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.788580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.788532    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.810655 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.810548    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.812623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.812580    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.910703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.890370    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:10:29.953231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.953198    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:10:29.953505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.953489    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:10:29.953682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.953653    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:10:29.953844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.953830    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:10:29.960206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.953955    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:10:29.960465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.960441    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:10:29.999691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:29.923836    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.083752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.064771    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:10:30.311873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.311812    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.334806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.333173    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.334984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.334952    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:10:30.335217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.335194    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:10:30.481417 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.481372    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.606741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.605801    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.645700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.644997    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.662695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.646226    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.836693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.833977    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.868744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:30.862783    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.092639 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.092575    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.094545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.094503    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.111085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.111046    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.112280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.112247    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.260589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.149381    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:10:31.322544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.322475    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:10:31.323773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.323724    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378391    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378479    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378528    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378593    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378705    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378750    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378868    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.378988    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.402028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.379049    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.432841 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.432790    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.462688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.433613    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.458265    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.527898    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.550686    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.551205    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.528711    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.566691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.553888    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:31.572937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.572881    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:31.602271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.602222    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "kube-api-access-697gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:31.610876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.610818    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.683691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.682559    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:10:31.684352 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.684306    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.694714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.686050    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.769692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.746165    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.786839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.786789    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:31.819768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.816055    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") pod \"472d6bf4-6c02-44ca-97de-27d02cdd1035\" (UID: \"472d6bf4-6c02-44ca-97de-27d02cdd1035\") "
Oct 30 22:10:31.826828 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:10:31.826798    4295 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/472d6bf4-6c02-44ca-97de-27d02cdd1035/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 30 22:10:31.827045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.827021    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo" pod "472d6bf4-6c02-44ca-97de-27d02cdd1035" (UID: "472d6bf4-6c02-44ca-97de-27d02cdd1035"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:31.832821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.832794    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.833033 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.833015    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.833190 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.833172    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697gr\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-kube-api-access-697gr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.836689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.833309    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.836855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.836836    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.836990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.836974    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472d6bf4-6c02-44ca-97de-27d02cdd1035-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.837122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.837108    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.837243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.837229    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472d6bf4-6c02-44ca-97de-27d02cdd1035-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.837362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.837348    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472d6bf4-6c02-44ca-97de-27d02cdd1035-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:31.861690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.861631    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.898175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:31.898128    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.130454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.128579    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.133895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.133836    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:32.151335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.151288    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:32.269830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.269795    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:10:32.327312 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.326920    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.331324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.331298    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:10:32.346506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.343853    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"
Oct 30 22:10:32.346506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.344391    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg" event={"ID":"472d6bf4-6c02-44ca-97de-27d02cdd1035","Type":"ContainerDied","Data":"f12500ca010bb7e66549dd72d0290e50479ca25423f43974fd437c1448d96ede"}
Oct 30 22:10:32.346506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.344438    4295 scope.go:122] "RemoveContainer" containerID="f463d3e326b0a52bc7eb9ed8f89135eda437d078ef37e18f188c069af619eb7d"
Oct 30 22:10:32.654480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.654439    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:10:32.663986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.663959    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:10:32.673791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.673755    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.740317 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.740289    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:10:32.759792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.759751    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.803686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.803588    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:10:32.805708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.805686    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:10:32.806280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.806233    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:10:32.808069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.808049    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:10:32.817457 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.817429    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:10:32.817628 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:32.817493    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:10:34.104473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:34.104433    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:10:34.108827 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:34.108798    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg"]
Oct 30 22:10:35.494877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:35.494836    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" path="/var/lib/kubelet/pods/472d6bf4-6c02-44ca-97de-27d02cdd1035/volumes"
Oct 30 22:10:38.552539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:38.552504    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:10:38.552960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:38.552715    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy" containerID="cri-o://5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342" gracePeriod=30
Oct 30 22:10:43.830867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.830794    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:10:43.944780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.944744    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.944780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.944787    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.944847    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.944884    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.944928    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945003    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945050    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:43.945245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945061    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945144    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945274    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:43.945413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945324    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\") pod \"34b1f297-079c-4d20-8a1b-12e45840ab9e\" (UID: \"34b1f297-079c-4d20-8a1b-12e45840ab9e\") "
Oct 30 22:10:43.945524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945430    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:43.945714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945583    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:43.945714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.945657    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:43.946249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.946223    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:43.946333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.946254    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:43.946333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.946269    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:43.946333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.946283    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b1f297-079c-4d20-8a1b-12e45840ab9e-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:43.946333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.946296    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:43.947378 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.947349    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:43.947480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.947431    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:43.947480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.947470    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:43.947555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:43.947487    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg" pod "34b1f297-079c-4d20-8a1b-12e45840ab9e" (UID: "34b1f297-079c-4d20-8a1b-12e45840ab9e"). InnerVolumeSpecName "kube-api-access-zrqwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.047896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.047859    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.047896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.047886    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.047896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.047898    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b1f297-079c-4d20-8a1b-12e45840ab9e-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.048199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.047906    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqwg\" (UniqueName: \"kubernetes.io/projected/34b1f297-079c-4d20-8a1b-12e45840ab9e-kube-api-access-zrqwg\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.434694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.434642    4295 generic.go:356] "Generic (PLEG): container finished" podID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerID="5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342" exitCode=0
Oct 30 22:10:44.434865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.434703    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" event={"ID":"34b1f297-079c-4d20-8a1b-12e45840ab9e","Type":"ContainerDied","Data":"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"}
Oct 30 22:10:44.434865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.434730    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"
Oct 30 22:10:44.434865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.434769    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk" event={"ID":"34b1f297-079c-4d20-8a1b-12e45840ab9e","Type":"ContainerDied","Data":"57aa4053ad9c0e2077d1a9659233c9037f4f7b3aa4fccdd8195e89e417cf2c4b"}
Oct 30 22:10:44.434865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.434791    4295 scope.go:122] "RemoveContainer" containerID="5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"
Oct 30 22:10:44.446488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.446462    4295 scope.go:122] "RemoveContainer" containerID="5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"
Oct 30 22:10:44.446834 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:10:44.446812    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342\": container with ID starting with 5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342 not found: ID does not exist" containerID="5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"
Oct 30 22:10:44.446923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.446845    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342"} err="failed to get container status \"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342\": rpc error: code = NotFound desc = could not find container \"5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342\": container with ID starting with 5e735441886cda2fd22afe6d3641903a52d35a6ddc9b3c8214d3a6a0a4e50342 not found: ID does not exist"
Oct 30 22:10:44.606590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.606554    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:10:44.657574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:44.657537    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk"]
Oct 30 22:10:45.494570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:45.494536    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" path="/var/lib/kubelet/pods/34b1f297-079c-4d20-8a1b-12e45840ab9e/volumes"
Oct 30 22:10:49.344875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.344834    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.345963    4295 cpu_manager.go:414] "containerMap: removing container" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy"
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.345983    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy"
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.346083    4295 cpu_manager.go:414] "containerMap: removing container" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy"
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.346090    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy"
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.346675    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="472d6bf4-6c02-44ca-97de-27d02cdd1035" containerName="istio-proxy"
Oct 30 22:10:49.347514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.346724    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b1f297-079c-4d20-8a1b-12e45840ab9e" containerName="istio-proxy"
Oct 30 22:10:49.351114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.351092    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.392672 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.392632    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:10:49.396441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.396426    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.429869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.429837    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.429983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.429963    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.429996    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430064 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430026    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430295 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430251    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430397    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430467    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430505    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430529    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwgs\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430558    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430620    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430739    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430778    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430838    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.430932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430868    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.430932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430895    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.431094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430930    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8trj\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.431094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.430964    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.449184 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.449160    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:10:49.501293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.501263    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:10:49.531548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531517    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.531740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531559    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.531740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531710    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.531831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531818    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.531880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531851    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.531916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531886    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.531955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531915    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwgs\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.531955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531946    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.532030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531975    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531995    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532060    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532106    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532159    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532180    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.532218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532204    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.532218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532210    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.532387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.531994    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.532522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532489    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532625    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532703    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532784    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8trj\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532798    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532856    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.532894    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.533234    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.533271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.533162    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.533821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.533780    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.534289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.534265    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.535101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.535082    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.536300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.536275    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.536378 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.536362    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.536440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.536419    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.596557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.596476    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8trj\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.598922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.598210    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwgs\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.599590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.599565    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:49.609437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.609412    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.664575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.664545    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:49.708837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:49.708798    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:50.098716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.098689    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:10:50.099647 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:10:50.099609    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaedf069_bcbb_4fb0_93ff_9fbe32a3b502.slice/crio-51de4a8468c352fec97e22d643cc5553479efd28f7b8b2f326c7c33d63b5202b WatchSource:0}: Error finding container 51de4a8468c352fec97e22d643cc5553479efd28f7b8b2f326c7c33d63b5202b: Status 404 returned error can't find the container with id 51de4a8468c352fec97e22d643cc5553479efd28f7b8b2f326c7c33d63b5202b
Oct 30 22:10:50.102168 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.102138    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.102229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.102204    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.102269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.102237    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.152212 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.152120    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:10:50.153156 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:10:50.153130    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf942fc11_7561_4f06_89bc_1c7cd3a3bdd1.slice/crio-c8f795a838b50764a59de48c6a2e743f8b9ad069fc845dcaf288e9651c9df39b WatchSource:0}: Error finding container c8f795a838b50764a59de48c6a2e743f8b9ad069fc845dcaf288e9651c9df39b: Status 404 returned error can't find the container with id c8f795a838b50764a59de48c6a2e743f8b9ad069fc845dcaf288e9651c9df39b
Oct 30 22:10:50.156355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.156317    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.156500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.156408    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.156500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.156453    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.480298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.480258    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" event={"ID":"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502","Type":"ContainerStarted","Data":"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"}
Oct 30 22:10:50.480862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.480329    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" event={"ID":"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502","Type":"ContainerStarted","Data":"51de4a8468c352fec97e22d643cc5553479efd28f7b8b2f326c7c33d63b5202b"}
Oct 30 22:10:50.482269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.482238    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" event={"ID":"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1","Type":"ContainerStarted","Data":"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"}
Oct 30 22:10:50.482418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.482296    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" event={"ID":"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1","Type":"ContainerStarted","Data":"c8f795a838b50764a59de48c6a2e743f8b9ad069fc845dcaf288e9651c9df39b"}
Oct 30 22:10:50.606993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.606939    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podStartSLOduration=1.6069223529999999 podStartE2EDuration="1.606922353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.603501182 +0000 UTC m=+1253.246095006" watchObservedRunningTime="2026-10-30 22:10:50.606922353 +0000 UTC m=+1253.249516142"
Oct 30 22:10:50.664916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.664878    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:50.711849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:50.711758    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:51.666189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:51.666152    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:51.666574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:51.666205    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:51.733690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:51.732813    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:51.733690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:51.732879    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:52.669943 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:52.669874    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:52.670398 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:52.669949    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:52.716690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:52.715018    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:52.716690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:52.715090    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:53.658713 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:10:53.658643    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:10:53.665209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:53.665182    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:53.665351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:53.665236    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:53.712467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:53.712359    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:53.712467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:53.712414    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:54.665806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:54.665768    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:54.665983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:54.665821    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:54.710079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:54.710043    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:54.710353 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:54.710324    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:55.665488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:55.665457    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:55.665893 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:55.665500    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:55.710689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:55.710048    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:55.710689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:55.710113    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:56.666050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:56.666014    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:10:56.666758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:56.666719    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 22:10:56.709498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:56.709457    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:10:56.709624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:56.709527    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:10:57.669582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.669555    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:57.670037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.670015    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:57.670843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.670823    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:10:57.714173 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.714154    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:57.714447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.714425    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:57.715251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.715234    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:10:57.851280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:10:57.851228    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podStartSLOduration=8.851213608 podStartE2EDuration="8.851213608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.741028174 +0000 UTC m=+1253.383621965" watchObservedRunningTime="2026-10-30 22:10:57.851213608 +0000 UTC m=+1260.493807400"
Oct 30 22:11:04.868891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:04.858043    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:04.868891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:04.859596    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:04.991717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:04.980353    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.166680 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.135175    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.166680 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.136385    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.166680 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.153729    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:05.282691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.281643    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.335207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.333779    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:11:05.459697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.456759    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.562579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.562529    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.772976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.772879    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.863813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.863759    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.871688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.869845    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.871688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.871051    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:11:05.871688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:05.871114    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.176890    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.177265    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.177507    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.178115    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.178169    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:11:06.185747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.178208    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:11:06.313699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.313022    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.313699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.313465    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:11:06.328688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.314490    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.373708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.338054    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.373708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.338464    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:11:06.480902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.464133    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.480902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.464480    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:11:06.719690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.666383    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.720487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.720464    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:11:06.801742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.799117    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.856689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.851113    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:11:06.912690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.880433    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.912690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.880804    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.955929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.955880    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.000690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:06.992746    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.242746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.233803    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.242746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.238790    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.242746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.239571    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.392689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.388523    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.430689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.430162    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:11:07.468847 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.468799    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.472685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.469426    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 22:11:07.512730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.512618    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 22:11:07.585691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.573779    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.793896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.793792    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:11:07.861612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.861549    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.886846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.886790    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.887549 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:07.887505    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.284021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.234766    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.284021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.239770    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.284021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.239871    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.432898 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.432840    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.437355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.437317    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.484161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.479034    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.586851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.586753    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.602889 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:11:08.602850    4295 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.031s"
Oct 30 22:11:08.626696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.616887    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:11:08.803911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.803873    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:08.826411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.813861    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.862755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.815985    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.875777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.833632    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded"
Oct 30 22:11:08.875777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.872477    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.888688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.887722    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.905810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.905769    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.993282 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.993247    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:08.993575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:08.993547    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.461689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.460839    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.465584 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.465547    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:11:09.481110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.465849    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.482335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.481806    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.483937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.482947    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:11:09.495778 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.495741    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.707761 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.683753    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.868759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.862345    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:09.868759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:09.862417    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.009560 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.009506    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.010338 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.010285    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.165746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.156818    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.166020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.165988    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.195349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.195137    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.514752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.506424    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.548753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.543314    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.548753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.543712    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.548753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.545776    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.690110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.690041    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.988742 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:11:10.985304    4295 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.499s"
Oct 30 22:11:10.988742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.985447    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:11:10.988742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:10.985485    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:11:11.011443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.011413    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:11:11.038805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.015403    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.146532 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.083694    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:11:11.146828 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.146807    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:11:11.147006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.146993    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:11:11.147937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.147912    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:11:11.148044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.147961    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:11:11.148044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.147994    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:11:11.148044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.148024    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:11:11.177500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:11.177474    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:11:23.609759 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:11:23.609723    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:11:28.057622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.057589    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:11:28.062023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.062005    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.116931 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.116891    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-643a82de-openshift-default-dockercfg-psjz6\""
Oct 30 22:11:28.158248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158220    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158277    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4l8\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158380    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158483    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158570    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158626    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158789    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.158908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158839    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.159005 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.158932    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.169484 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.169458    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.265931    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4l8\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.265994    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266082    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266119    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266173    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266264    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266311    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266435    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.266528    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.271689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.267032    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.289726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.279138    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.289726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.279467    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.289726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.280148    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.289726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.280748    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.289726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.280985    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.330318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.323086    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.387579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.348807    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4l8\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.391191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.391141    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\") pod \"e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") " pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:28.675689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:28.675577    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:29.111257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.111228    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:11:29.145236 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:11:29.145208    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9111e2_a23a_4a6e_813f_57113be2996f.slice/crio-6f5c3a7f74e5769718f181f03a1c6d3fd5e701f5f91bc753df5f50f5714b4acd WatchSource:0}: Error finding container 6f5c3a7f74e5769718f181f03a1c6d3fd5e701f5f91bc753df5f50f5714b4acd: Status 404 returned error can't find the container with id 6f5c3a7f74e5769718f181f03a1c6d3fd5e701f5f91bc753df5f50f5714b4acd
Oct 30 22:11:29.147771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.147740    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:29.147841 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.147806    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:29.147841 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.147835    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:29.401483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.401404    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerStarted","Data":"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"}
Oct 30 22:11:29.401483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.401454    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerStarted","Data":"6f5c3a7f74e5769718f181f03a1c6d3fd5e701f5f91bc753df5f50f5714b4acd"}
Oct 30 22:11:29.676625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:29.676441    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:30.681322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:30.681290    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:30.828484 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:30.828437    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podStartSLOduration=2.828424068 podStartE2EDuration="2.828424068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.610327432 +0000 UTC m=+1292.252921222" watchObservedRunningTime="2026-10-30 22:11:30.828424068 +0000 UTC m=+1293.471017856"
Oct 30 22:11:31.417054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:31.417025    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:31.417968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:31.417950    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:11:35.996690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:35.956317    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:11:36.020027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.013873    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:11:36.020027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.013930    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:11:36.026702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.025861    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.070906 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.070858    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:11:36.071301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.071279    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.105284 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.100018    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-08235cc9\""
Oct 30 22:11:36.158111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.131222    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-08235cc9-dockercfg-lqwhq\""
Oct 30 22:11:36.158111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.141799    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.158111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.141930    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.158111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.141972    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.158111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.142068    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkmr\" (UniqueName: \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.258494    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkmr\" (UniqueName: \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.258740    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.258849    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.258888    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.260020    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.293524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.272712    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.404748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.390595    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\") pod \"payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") " pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.420690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.411960    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkmr\" (UniqueName: \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\") pod \"payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.421178 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.420928    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:36.485804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.482971    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:36.898650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.895912    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:11:36.898650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.896754    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:36.919880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.919780    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:36.958705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:36.940773    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.035586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.004818    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.257801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.257753    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:37.470845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.462358    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.563743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.558189    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.645773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.627292    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.702397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.702342    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.917967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.915864    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.917967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.917385    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:11:37.945066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.945020    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:37.946345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.946322    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:11:37.969070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.946522    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:11:37.970398 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:37.970374    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:11:38.269834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.268940    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:38.273302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.273271    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:11:38.275235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.275149    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.442867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.442825    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.575255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.575154    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.630787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.630744    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.746691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.635061    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:11:38.746691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.630920    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.746691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.645553    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:11:38.840934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.795449    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.840934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.811521    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:11:38.840934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.838571    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.840934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.840757    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:38.840934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.840812    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.916911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.916852    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.969960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.969831    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.989690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.969735    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.989690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:38.971018    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.021023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.020884    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:39.021023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.020948    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.202613 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:11:39.202579    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988d0b92_7666_4e98_ba42_7381649a0db5.slice/crio-34b2855d4277b52e16e5d81bc94b236abf47e4fb3aea7d837012ad0b776e5bc7 WatchSource:0}: Error finding container 34b2855d4277b52e16e5d81bc94b236abf47e4fb3aea7d837012ad0b776e5bc7: Status 404 returned error can't find the container with id 34b2855d4277b52e16e5d81bc94b236abf47e4fb3aea7d837012ad0b776e5bc7
Oct 30 22:11:39.235468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.235424    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:11:39.301096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.301045    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.304686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.301925    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:11:39.304686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.302916    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:11:39.653744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.653329    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.662431 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.662369    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:11:39.817223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.771525    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.865691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.840856    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.865691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.841407    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.865691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:39.841979    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 22:11:40.052697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.046978    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.052697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.047090    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.114701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.113705    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.114701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.113792    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.438815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.362812    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.438815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.363210    4295 trace.go:236] Trace[1105300788]: "Calculate volume metrics of dshm for pod e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" (30-Oct-2026 22:11:39.324) (total time: 1039ms):
Oct 30 22:11:40.438815 ip-10-0-135-59 kubenswrapper[4295]: Trace[1105300788]: [1.039140051s] [1.039140051s] END
Oct 30 22:11:40.438815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.363344    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.438815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.421942    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.520689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.498160    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.520689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.498865    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.658749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.630270    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.669689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.665763    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.861690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.859123    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.861690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.859298    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.876843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.876199    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.892567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.892530    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.892890 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.892858    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.896686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.893112    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.973153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.899063    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.982339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.982267    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.986355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:40.985333    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:11:41.131953    4295 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.362s"
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.132031    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.132078    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.132112    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.132185    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" event={"ID":"4bcc81dd-b855-43cb-afdc-690f7e931979","Type":"ContainerStarted","Data":"2c375c90c296554b29caa3f6ede46231fd9f41f468c84fdf9869d89a477528d8"}
Oct 30 22:11:41.132689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.132229    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:11:41.252500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.252458    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:11:41.252843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.252811    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:11:41.358819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.358775    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 22:11:41.365303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.365277    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:11:41.375710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.372023    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:11:41.376793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.376771    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:11:41.422370 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.422327    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.512738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.509302    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.537243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.518846    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.683416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.682375    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:11:41.730141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.730096    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.738083 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.738047    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:41.738327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.738298    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:41.889236    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:11:42.083071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.083040    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:11:42.133184 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.133156    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:11:42.133507 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.133489    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.173126 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.173093    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:11:42.199508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.197540    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:11:42.209974 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.209940    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:11:42.215819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.215460    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:11:42.219866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.218901    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:11:42.225615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.225583    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:11:42.226283 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.226266    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-643a82de-dockercfg-p9ztg\""
Oct 30 22:11:42.229784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.229760    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-643a82de\""
Oct 30 22:11:42.237172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.237146    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" event={"ID":"4bcc81dd-b855-43cb-afdc-690f7e931979","Type":"ContainerStarted","Data":"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"}
Oct 30 22:11:42.237446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.237429    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:42.239441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.239417    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" event={"ID":"988d0b92-7666-4e98-ba42-7381649a0db5","Type":"ContainerStarted","Data":"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"}
Oct 30 22:11:42.239625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.239596    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" event={"ID":"988d0b92-7666-4e98-ba42-7381649a0db5","Type":"ContainerStarted","Data":"34b2855d4277b52e16e5d81bc94b236abf47e4fb3aea7d837012ad0b776e5bc7"}
Oct 30 22:11:42.314183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.314097    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.314765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.314639    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclxw\" (UniqueName: \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.415862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.415821    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.416009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.415896    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclxw\" (UniqueName: \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.416597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.416575    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.480483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.480449    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclxw\" (UniqueName: \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\") pod \"payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.537469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.537434    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:42.971221 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:42.971190    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:11:42.972393 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:11:42.972367    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfaf1899_04bf_433d_9c2b_61b7575e7d4c.slice/crio-cf404c81118764dbf935ac9a094cf9baf9d00fe7ff94df61c06edc0ce5c83398 WatchSource:0}: Error finding container cf404c81118764dbf935ac9a094cf9baf9d00fe7ff94df61c06edc0ce5c83398: Status 404 returned error can't find the container with id cf404c81118764dbf935ac9a094cf9baf9d00fe7ff94df61c06edc0ce5c83398
Oct 30 22:11:43.247261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.247167    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" event={"ID":"dfaf1899-04bf-433d-9c2b-61b7575e7d4c","Type":"ContainerStarted","Data":"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"}
Oct 30 22:11:43.247261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.247242    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" event={"ID":"dfaf1899-04bf-433d-9c2b-61b7575e7d4c","Type":"ContainerStarted","Data":"cf404c81118764dbf935ac9a094cf9baf9d00fe7ff94df61c06edc0ce5c83398"}
Oct 30 22:11:43.247525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.247507    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:43.247593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.247549    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:43.528513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.528415    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" podStartSLOduration=8.528398419 podStartE2EDuration="8.528398419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:43.52612711 +0000 UTC m=+1306.168720900" watchObservedRunningTime="2026-10-30 22:11:43.528398419 +0000 UTC m=+1306.170992211"
Oct 30 22:11:43.779739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.773026    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" podStartSLOduration=8.773004892 podStartE2EDuration="8.773004892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:43.771137802 +0000 UTC m=+1306.413731606" watchObservedRunningTime="2026-10-30 22:11:43.773004892 +0000 UTC m=+1306.415598682"
Oct 30 22:11:43.988442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:43.986158    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" podStartSLOduration=4.986136147 podStartE2EDuration="4.986136147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:43.975970004 +0000 UTC m=+1306.618563795" watchObservedRunningTime="2026-10-30 22:11:43.986136147 +0000 UTC m=+1306.628729938"
Oct 30 22:11:53.250451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:53.250416    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:11:53.608619 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:11:53.608542    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:11:54.256833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:54.256805    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:11:54.257251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:54.256869    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:11:57.723852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:11:57.723820    4295 scope.go:122] "RemoveContainer" containerID="9eb2296dcf1e5b80fb5b10ed4ef5a7ba3734b7ab218d05887fb5be58af30fd71"
Oct 30 22:12:12.667581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.667552    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:12:12.673127 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.673104    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.707162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.707130    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj88x\" (UniqueName: \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.707274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.707168    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.707274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.707193    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.707417 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.707397    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.778407 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.778376    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4cce86f1-serving-cert\""
Oct 30 22:12:12.781817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.781797    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:12:12.809149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.809115    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.809321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.809288    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.809543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.809524    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj88x\" (UniqueName: \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.809606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.809572    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.810531 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:12:12.810511    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-serving-cert: secret "maas-api-e2e-migrate-4cce86f1-serving-cert" not found
Oct 30 22:12:12.810752 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:12:12.810737    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls podName:bafbe951-ceb8-4e7b-b2c3-3b03647b6208 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:13.310715933 +0000 UTC m=+1335.953309712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls") pod "maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" (UID: "bafbe951-ceb8-4e7b-b2c3-3b03647b6208") : secret "maas-api-e2e-migrate-4cce86f1-serving-cert" not found
Oct 30 22:12:12.810878 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.810771    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.813152 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.813124    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:12.873862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:12.873833    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj88x\" (UniqueName: \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:13.325082 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:13.325051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:13.327839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:13.327812    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") pod \"maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:13.586258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:13.586181    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:13.965510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:13.965478    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:12:13.966252 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:12:13.966228    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafbe951_ceb8_4e7b_b2c3_3b03647b6208.slice/crio-c43dd60959875e9173cb759585a9a3ba78af5206430d1aa1d44e6f1fccd63cd5 WatchSource:0}: Error finding container c43dd60959875e9173cb759585a9a3ba78af5206430d1aa1d44e6f1fccd63cd5: Status 404 returned error can't find the container with id c43dd60959875e9173cb759585a9a3ba78af5206430d1aa1d44e6f1fccd63cd5
Oct 30 22:12:13.967649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:13.967632    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:12:14.448889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:14.448847    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" event={"ID":"bafbe951-ceb8-4e7b-b2c3-3b03647b6208","Type":"ContainerStarted","Data":"c43dd60959875e9173cb759585a9a3ba78af5206430d1aa1d44e6f1fccd63cd5"}
Oct 30 22:12:15.457758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:15.457725    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" event={"ID":"bafbe951-ceb8-4e7b-b2c3-3b03647b6208","Type":"ContainerStarted","Data":"061952593ece01d168dfdaca0a3eb0d1179a72068eab6b37d624c44855d85d87"}
Oct 30 22:12:15.458160 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:15.457814    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:15.592875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:15.592822    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" podStartSLOduration=2.830035685 podStartE2EDuration="3.592808121s" totalImagesPullingTime="762.772436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:15.589533195 +0000 UTC m=+1338.232126985" watchObservedRunningTime="2026-10-30 22:12:15.592808121 +0000 UTC m=+1338.235401911"
Oct 30 22:12:17.561529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.561497    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:12:17.565355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.565338    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.621522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.621497    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-08235cc9-serving-cert\""
Oct 30 22:12:17.626342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.626316    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:12:17.700387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.700352    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.700543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.700484    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlckt\" (UniqueName: \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.700612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.700593    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.700713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.700694    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.841744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.820064    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.842075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.842046    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlckt\" (UniqueName: \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.842328 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.842304    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.842495 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.842482    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.859682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.858720    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.872421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.872386    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.878764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.875194    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:17.926689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:17.919436    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlckt\" (UniqueName: \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\") pod \"maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:18.179263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:18.179237    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:18.560555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:18.560531    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:12:18.561003 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:12:18.560975    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a01db29_484f_46cc_9ef1_3ee984d81069.slice/crio-c8efd446c49a2fef633b01450d35d34fafb6149f63a4aafd0476e6146d8f6d18 WatchSource:0}: Error finding container c8efd446c49a2fef633b01450d35d34fafb6149f63a4aafd0476e6146d8f6d18: Status 404 returned error can't find the container with id c8efd446c49a2fef633b01450d35d34fafb6149f63a4aafd0476e6146d8f6d18
Oct 30 22:12:19.494217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:19.494182    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" event={"ID":"8a01db29-484f-46cc-9ef1-3ee984d81069","Type":"ContainerStarted","Data":"c8efd446c49a2fef633b01450d35d34fafb6149f63a4aafd0476e6146d8f6d18"}
Oct 30 22:12:20.494149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:20.494117    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" event={"ID":"8a01db29-484f-46cc-9ef1-3ee984d81069","Type":"ContainerStarted","Data":"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"}
Oct 30 22:12:20.494339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:20.494247    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:20.617304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:20.617254    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" podStartSLOduration=2.7157469499999998 podStartE2EDuration="3.617240458s" totalImagesPullingTime="901.493508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:20.614983751 +0000 UTC m=+1343.257577541" watchObservedRunningTime="2026-10-30 22:12:20.617240458 +0000 UTC m=+1343.259834246"
Oct 30 22:12:21.470433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:21.470402    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:12:23.608110 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:12:23.608073    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:12:24.663169 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.663140    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:12:24.667307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.667287    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.756935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.756902    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnm2d\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.756966    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757045    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757124    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757142    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757299    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757390 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757332    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757425    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.757508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.757452    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858530    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858572    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858688    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858722    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858836    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.858904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858869    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858931    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.858963    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859013    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859065    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnm2d\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859281 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859256    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859275    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859319    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.859586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.859567    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.861213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.861185    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.861319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.861227    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.879765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.879738    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:12:24.922607 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.922545    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnm2d\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.922607 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.922553    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9hjxv\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:24.981139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:24.981107    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:25.473329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:25.473304    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:12:25.473943 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:12:25.473922    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad509fd_8dfd_47f9_847f_b07248210066.slice/crio-536f6ac535830781988202d4ccd01b77a873850bd0e075c569ef266bc242ae98 WatchSource:0}: Error finding container 536f6ac535830781988202d4ccd01b77a873850bd0e075c569ef266bc242ae98: Status 404 returned error can't find the container with id 536f6ac535830781988202d4ccd01b77a873850bd0e075c569ef266bc242ae98
Oct 30 22:12:25.476245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:25.476197    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.476352 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:25.476287    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.476352 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:25.476333    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.526452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:25.526294    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" event={"ID":"3ad509fd-8dfd-47f9-847f-b07248210066","Type":"ContainerStarted","Data":"536f6ac535830781988202d4ccd01b77a873850bd0e075c569ef266bc242ae98"}
Oct 30 22:12:26.514791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.514756    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:12:26.535271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.535242    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" event={"ID":"3ad509fd-8dfd-47f9-847f-b07248210066","Type":"ContainerStarted","Data":"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"}
Oct 30 22:12:26.817106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.816999    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podStartSLOduration=2.816982915 podStartE2EDuration="2.816982915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:26.814129277 +0000 UTC m=+1349.456723102" watchObservedRunningTime="2026-10-30 22:12:26.816982915 +0000 UTC m=+1349.459576704"
Oct 30 22:12:26.981534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.981504    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:26.982705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.982681    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:26.982814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:26.982733    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:27.981921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:27.981873    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:27.982303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:27.981968    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:28.114402 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.114358    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:12:28.118776 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.118750    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.165739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.165713    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:12:28.173899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.173878    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-643a82de-serving-cert\""
Oct 30 22:12:28.240430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.240333    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.240581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.240468    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.240678 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.240638    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.240892 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.240816    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnccl\" (UniqueName: \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.349438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.349389    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.354327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.354293    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnccl\" (UniqueName: \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.354510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.354493    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.354604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.354588    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.356714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.356606    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.373337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.373302    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.397718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.393021    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.478690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.473880    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnccl\" (UniqueName: \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\") pod \"maas-api-e2e-ait-643a82de-66d65fcb74-24twt\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.732617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.732575    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:28.982396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.982367    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:28.982804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:28.982421    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:29.156016 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:12:29.155979    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf1a254_d564_40d0_b6f5_1edac676ed33.slice/crio-428cb760a695817ee5a3f0a15b8d487c1098026e6a6af14cd2e1f6cce341985a WatchSource:0}: Error finding container 428cb760a695817ee5a3f0a15b8d487c1098026e6a6af14cd2e1f6cce341985a: Status 404 returned error can't find the container with id 428cb760a695817ee5a3f0a15b8d487c1098026e6a6af14cd2e1f6cce341985a
Oct 30 22:12:29.156293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:29.156271    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:12:29.713234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:29.713193    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" event={"ID":"fcf1a254-d564-40d0-b6f5-1edac676ed33","Type":"ContainerStarted","Data":"428cb760a695817ee5a3f0a15b8d487c1098026e6a6af14cd2e1f6cce341985a"}
Oct 30 22:12:30.003944 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.003859    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:30.026706 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.022228    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:30.756704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.756377    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:30.800698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.799002    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" event={"ID":"fcf1a254-d564-40d0-b6f5-1edac676ed33","Type":"ContainerStarted","Data":"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"}
Oct 30 22:12:30.800698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.800040    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:30.849766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.849071    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:30.849766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.849604    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:30.974688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.966397    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" podStartSLOduration=1.944369172 podStartE2EDuration="2.966375785s" totalImagesPullingTime="1.022006613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:30.94591926 +0000 UTC m=+1353.588513053" watchObservedRunningTime="2026-10-30 22:12:30.966375785 +0000 UTC m=+1353.608969573"
Oct 30 22:12:31.003950 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:30.982384    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:31.030953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.030913    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:31.031544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.004543    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.096709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.095747    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.157741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.149780    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.303989 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.303942    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.368070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.368026    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.460789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.460745    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.612337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.612295    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.756972 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.756871    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.758182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.758154    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:12:31.856688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.854678    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.856863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.856719    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.860595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.857514    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:12:31.860595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.857584    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:12:31.899628 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.895070    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.948839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:31.948043    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:12:32.032214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.032118    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.033200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.033177    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:12:32.171858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.169833    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.171858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.170207    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:12:32.289318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.286744    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:32.289318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.286794    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:32.310324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.310280    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:12:32.310613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.310573    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:12:32.318684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.312385    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:12:32.398811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.398773    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.399380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.399361    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:12:32.627952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.627862    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:12:32.635690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.628527    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:12:32.640696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.633047    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.650553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.648059    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:12:32.650915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.650897    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:12:32.684259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.658917    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:12:32.743783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.742872    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:12:32.743783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.743698    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:12:32.771222 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.771174    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.780164 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.779893    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:12:32.832883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.832838    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:12:32.838214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.838155    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:12:32.853713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.844024    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:12:32.981949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.981919    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:32.982119 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:32.981963    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:33.982511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:33.982475    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:33.983011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:33.982537    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:34.981741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:34.981705    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:12:34.981925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:34.981761    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 22:12:36.187773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:36.187745    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:36.188283 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:36.188258    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:36.189052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:36.189028    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:12:37.878054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:37.878022    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:12:53.610024 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:12:53.609983    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:12:54.777938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.777905    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:12:54.782620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.782597    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863288 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863259    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863312    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78724\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863341    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863426    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863547    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863713    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863760    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863857 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863827    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.863857 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.863852    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.892715 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.892652    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.982727    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.982844    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78724\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.982897    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.982930    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.983024    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.983140    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.983183    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.983217    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.983245    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.984136    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.985686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.984456    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.993073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.992947    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.999016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.993255    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.999016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.993894    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:54.999016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:54.996914    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:55.049034 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.048947    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:55.053322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.053281    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78724\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:55.083695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.079788    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8kz9s\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:55.096557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.096210    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:55.537888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.537861    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:12:55.538530 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:12:55.538506    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5275837_506f_463f_9ed4_a1c77ee76482.slice/crio-92fd6991932b275b9ea8c15e0987d98f184de2b6e439153cc102ae4214d3bf68 WatchSource:0}: Error finding container 92fd6991932b275b9ea8c15e0987d98f184de2b6e439153cc102ae4214d3bf68: Status 404 returned error can't find the container with id 92fd6991932b275b9ea8c15e0987d98f184de2b6e439153cc102ae4214d3bf68
Oct 30 22:12:55.541021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.540988    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:55.541086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.541051    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:55.541086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:55.541080    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:56.109804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:56.109518    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" event={"ID":"c5275837-506f-463f-9ed4-a1c77ee76482","Type":"ContainerStarted","Data":"de77d2039e96297941af6f9d2fc0ba22343be2e0702c831cd7e78e5c4e17055d"}
Oct 30 22:12:56.109804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:56.109588    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" event={"ID":"c5275837-506f-463f-9ed4-a1c77ee76482","Type":"ContainerStarted","Data":"92fd6991932b275b9ea8c15e0987d98f184de2b6e439153cc102ae4214d3bf68"}
Oct 30 22:12:56.237983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:56.237922    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podStartSLOduration=2.23790492 podStartE2EDuration="2.23790492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:56.235972533 +0000 UTC m=+1378.878566323" watchObservedRunningTime="2026-10-30 22:12:56.23790492 +0000 UTC m=+1378.880498709"
Oct 30 22:12:57.109689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:57.108992    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:12:57.114105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:57.114073    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.114585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:57.114171    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:12:58.097696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:58.097009    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.097696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:58.097071    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:12:59.109689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:59.108802    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.109689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:12:59.108867    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:13:00.101690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:00.097996    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.101690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:00.098071    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:13:01.112725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:01.102014    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:13:01.112725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:01.102089    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:13:02.097002 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:02.096956    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:13:02.097221 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:02.097013    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 22:13:03.900125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:03.900096    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:13:03.900846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:03.900820    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:13:03.901583 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:03.901563    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:13:11.324257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.324019    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.708496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.706871    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.708496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.707878    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.755687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.754854    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:13:11.755994 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.755961    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.789991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.789924    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.837347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.836066    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.882092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.851753    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:13:11.921709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.921641    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.950858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:11.950809    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.755336 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.755292    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.756318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.756289    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.757140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.757119    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:13:12.766029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.765995    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:13:12.772555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.772508    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.883246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.883195    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.884066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.884044    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:13:12.892365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.892334    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.901741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.901713    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:13:12.960690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.959820    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.960690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:12.960400    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:13:13.335795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.335252    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.661995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.661895    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.693544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.692762    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.810689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.789006    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.810689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.790285    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.810689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.792149    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:13:13.822886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.822828    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.858410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.858366    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:13:13.903714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.892221    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:13:13.903714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.897738    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.904106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.904056    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:13:13.973851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:13.973806    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.317842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.296686    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:13:14.700785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.700741    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.701186 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.701165    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:13:14.793205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.793164    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.793707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.793678    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.794471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.794450    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:13:14.906786 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.906724    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.937702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.924271    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:13:14.937702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.925307    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.937702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.925572    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.945487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.944776    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn" podUID="69d373d5-5760-4d37-8351-8668586c8f88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.951577 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.951464    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.984719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:14.974815    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:13:15.352737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.343506    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:13:15.352737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.350809    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:13:15.469693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.466739    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:13:15.481692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.479698    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:13:15.631256 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.631172    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn"
Oct 30 22:13:15.648194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.648164    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:13:15.650432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.650408    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:13:15.660718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.659337    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:13:15.665183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:15.665155    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:13:20.684545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.684512    4295 generic.go:356] "Generic (PLEG): container finished" podID="de9111e2-a23a-4a6e-813f-57113be2996f" containerID="091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d" exitCode=0
Oct 30 22:13:20.684987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.684586    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerDied","Data":"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"}
Oct 30 22:13:20.685126 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.685109    4295 scope.go:122] "RemoveContainer" containerID="091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"
Oct 30 22:13:20.686864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.686835    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:20.686931 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.686909    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:20.686968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:20.686940    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:21.718465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:21.718428    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerStarted","Data":"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"}
Oct 30 22:13:22.676260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:22.676213    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:13:22.676553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:22.676348    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:13:22.676553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:22.676399    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 22:13:23.685574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:23.685538    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:13:23.686161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:23.686130    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 22:13:23.823774 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:13:23.823735    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:13:24.677036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:24.677003    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:13:24.677249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:24.677055    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 22:13:25.676524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:25.676482    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:13:25.676972 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:25.676546    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 22:13:26.676252 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:26.676209    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:13:26.676425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:26.676275    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 22:13:27.680430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:27.680402    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:13:27.680905 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:27.680888    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:13:27.681458 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:27.681439    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:13:39.085040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:39.085006    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:13:39.213893 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:39.213861    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:13:44.085539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.085467    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" podUID="4bcc81dd-b855-43cb-afdc-690f7e931979" containerName="payload-processing" containerID="cri-o://bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb" gracePeriod=25
Oct 30 22:13:44.214553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.214501    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" containerName="payload-pre-processing" containerID="cri-o://c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384" gracePeriod=25
Oct 30 22:13:44.254712 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:13:44.254685    4295 logging.go:55] [core] [Channel #1098 SubChannel #1100]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.76:50052", ServerName: "10.133.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.76:50052: connect: connection refused"
Oct 30 22:13:44.383762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.383730    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:13:44.519365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.518570    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\") pod \"4bcc81dd-b855-43cb-afdc-690f7e931979\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") "
Oct 30 22:13:44.519365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.518902    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\") pod \"4bcc81dd-b855-43cb-afdc-690f7e931979\" (UID: \"4bcc81dd-b855-43cb-afdc-690f7e931979\") "
Oct 30 22:13:44.519365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.519295    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume" pod "4bcc81dd-b855-43cb-afdc-690f7e931979" (UID: "4bcc81dd-b855-43cb-afdc-690f7e931979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:44.519758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.519733    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:13:44.520962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.520934    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk" pod "4bcc81dd-b855-43cb-afdc-690f7e931979" (UID: "4bcc81dd-b855-43cb-afdc-690f7e931979"). InnerVolumeSpecName "kube-api-access-z7vnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:44.620857 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.620765    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\" (UniqueName: \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\") pod \"988d0b92-7666-4e98-ba42-7381649a0db5\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") "
Oct 30 22:13:44.621020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.620987    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\") pod \"988d0b92-7666-4e98-ba42-7381649a0db5\" (UID: \"988d0b92-7666-4e98-ba42-7381649a0db5\") "
Oct 30 22:13:44.621443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.621368    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume" pod "988d0b92-7666-4e98-ba42-7381649a0db5" (UID: "988d0b92-7666-4e98-ba42-7381649a0db5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:44.621634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.621513    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988d0b92-7666-4e98-ba42-7381649a0db5-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.621634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.621528    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bcc81dd-b855-43cb-afdc-690f7e931979-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.621634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.621537    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/4bcc81dd-b855-43cb-afdc-690f7e931979-kube-api-access-z7vnk\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.623027 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.623004    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr" pod "988d0b92-7666-4e98-ba42-7381649a0db5" (UID: "988d0b92-7666-4e98-ba42-7381649a0db5"). InnerVolumeSpecName "kube-api-access-bfkmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:44.722527 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.722480    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkmr\" (UniqueName: \"kubernetes.io/projected/988d0b92-7666-4e98-ba42-7381649a0db5-kube-api-access-bfkmr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.989403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.989368    4295 generic.go:356] "Generic (PLEG): container finished" podID="988d0b92-7666-4e98-ba42-7381649a0db5" containerID="c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384" exitCode=0
Oct 30 22:13:44.989593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.989407    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"
Oct 30 22:13:44.989593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.989439    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" event={"ID":"988d0b92-7666-4e98-ba42-7381649a0db5","Type":"ContainerDied","Data":"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"}
Oct 30 22:13:44.989593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.989495    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" event={"ID":"988d0b92-7666-4e98-ba42-7381649a0db5","Type":"ContainerDied","Data":"34b2855d4277b52e16e5d81bc94b236abf47e4fb3aea7d837012ad0b776e5bc7"}
Oct 30 22:13:44.989593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.989524    4295 scope.go:122] "RemoveContainer" containerID="c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"
Oct 30 22:13:44.991089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.991054    4295 generic.go:356] "Generic (PLEG): container finished" podID="4bcc81dd-b855-43cb-afdc-690f7e931979" containerID="bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb" exitCode=0
Oct 30 22:13:44.991194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.991093    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"
Oct 30 22:13:44.991256 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.991186    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" event={"ID":"4bcc81dd-b855-43cb-afdc-690f7e931979","Type":"ContainerDied","Data":"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"}
Oct 30 22:13:44.991304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:44.991294    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7" event={"ID":"4bcc81dd-b855-43cb-afdc-690f7e931979","Type":"ContainerDied","Data":"2c375c90c296554b29caa3f6ede46231fd9f41f468c84fdf9869d89a477528d8"}
Oct 30 22:13:45.000741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.000725    4295 scope.go:122] "RemoveContainer" containerID="c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"
Oct 30 22:13:45.001046 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:13:45.001021    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384\": container with ID starting with c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384 not found: ID does not exist" containerID="c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"
Oct 30 22:13:45.001133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.001057    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384"} err="failed to get container status \"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384\": rpc error: code = NotFound desc = could not find container \"c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384\": container with ID starting with c00a3439c3efc6733b9d9aad8c8da6cd01252ad8b37d964ad78bb005c49f0384 not found: ID does not exist"
Oct 30 22:13:45.001133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.001079    4295 scope.go:122] "RemoveContainer" containerID="bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"
Oct 30 22:13:45.009931 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.009906    4295 scope.go:122] "RemoveContainer" containerID="bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"
Oct 30 22:13:45.010217 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:13:45.010196    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb\": container with ID starting with bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb not found: ID does not exist" containerID="bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"
Oct 30 22:13:45.010275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.010225    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb"} err="failed to get container status \"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb\": rpc error: code = NotFound desc = could not find container \"bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb\": container with ID starting with bc1e5da1786c011d43a80a5de30e76c3af0c9d761009ea82904c737bbc5866cb not found: ID does not exist"
Oct 30 22:13:45.161207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.161170    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:13:45.211500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.211467    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g"]
Oct 30 22:13:45.255471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.255374    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.76:50052\" within 1s: context deadline exceeded"
Oct 30 22:13:45.255646 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:13:45.255399    4295 logging.go:55] [core] [Channel #1098 SubChannel #1100]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.76:50052", ServerName: "10.133.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.76:50052: operation was canceled"
Oct 30 22:13:45.276123 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.276094    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:13:45.281948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.281922    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7"]
Oct 30 22:13:45.496170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.496134    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcc81dd-b855-43cb-afdc-690f7e931979" path="/var/lib/kubelet/pods/4bcc81dd-b855-43cb-afdc-690f7e931979/volumes"
Oct 30 22:13:45.496656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:45.496634    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" path="/var/lib/kubelet/pods/988d0b92-7666-4e98-ba42-7381649a0db5/volumes"
Oct 30 22:13:52.029117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:52.029073    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:13:53.735299 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:13:53.735238    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:13:57.029444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.029382    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" podUID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" containerName="payload-pre-processing" containerID="cri-o://e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e" gracePeriod=25
Oct 30 22:13:57.318314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.318280    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:13:57.380574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.380548    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\") pod \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") "
Oct 30 22:13:57.380773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.380608    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\" (UniqueName: \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\") pod \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\" (UID: \"dfaf1899-04bf-433d-9c2b-61b7575e7d4c\") "
Oct 30 22:13:57.380976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.380955    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume" pod "dfaf1899-04bf-433d-9c2b-61b7575e7d4c" (UID: "dfaf1899-04bf-433d-9c2b-61b7575e7d4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:57.381265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.381245    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.382768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.382748    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw" pod "dfaf1899-04bf-433d-9c2b-61b7575e7d4c" (UID: "dfaf1899-04bf-433d-9c2b-61b7575e7d4c"). InnerVolumeSpecName "kube-api-access-mclxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.482564 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:57.482528    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclxw\" (UniqueName: \"kubernetes.io/projected/dfaf1899-04bf-433d-9c2b-61b7575e7d4c-kube-api-access-mclxw\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:58.132795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.132647    4295 generic.go:356] "Generic (PLEG): container finished" podID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" containerID="e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e" exitCode=0
Oct 30 22:13:58.132795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.132715    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"
Oct 30 22:13:58.132795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.132739    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" event={"ID":"dfaf1899-04bf-433d-9c2b-61b7575e7d4c","Type":"ContainerDied","Data":"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"}
Oct 30 22:13:58.133396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.132804    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7" event={"ID":"dfaf1899-04bf-433d-9c2b-61b7575e7d4c","Type":"ContainerDied","Data":"cf404c81118764dbf935ac9a094cf9baf9d00fe7ff94df61c06edc0ce5c83398"}
Oct 30 22:13:58.133396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.132837    4295 scope.go:122] "RemoveContainer" containerID="e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"
Oct 30 22:13:58.143205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.143187    4295 scope.go:122] "RemoveContainer" containerID="e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"
Oct 30 22:13:58.143504 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:13:58.143483    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e\": container with ID starting with e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e not found: ID does not exist" containerID="e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"
Oct 30 22:13:58.143597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.143515    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e"} err="failed to get container status \"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e\": rpc error: code = NotFound desc = could not find container \"e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e\": container with ID starting with e0a638bd318ede952435624c27e68ca86e432ef6c0e9ee14542e3f46c73a1e8e not found: ID does not exist"
Oct 30 22:13:58.325147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.325115    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:13:58.375021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:58.374991    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7"]
Oct 30 22:13:59.494596 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:13:59.494565    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" path="/var/lib/kubelet/pods/dfaf1899-04bf-433d-9c2b-61b7575e7d4c/volumes"
Oct 30 22:14:13.309880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:13.309848    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:14:13.310318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:13.310079    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" podUID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" containerName="maas-api" containerID="cri-o://061952593ece01d168dfdaca0a3eb0d1179a72068eab6b37d624c44855d85d87" gracePeriod=30
Oct 30 22:14:14.245941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.245852    4295 generic.go:356] "Generic (PLEG): container finished" podID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" containerID="061952593ece01d168dfdaca0a3eb0d1179a72068eab6b37d624c44855d85d87" exitCode=0
Oct 30 22:14:14.246235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.246211    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" event={"ID":"bafbe951-ceb8-4e7b-b2c3-3b03647b6208","Type":"ContainerDied","Data":"061952593ece01d168dfdaca0a3eb0d1179a72068eab6b37d624c44855d85d87"}
Oct 30 22:14:14.305854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.305822    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:14:14.411112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.411074    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\" (UniqueName: \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\") pod \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") "
Oct 30 22:14:14.411623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.411195    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\") pod \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") "
Oct 30 22:14:14.411623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.411261    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\") pod \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") "
Oct 30 22:14:14.411623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.411402    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") pod \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\" (UID: \"bafbe951-ceb8-4e7b-b2c3-3b03647b6208\") "
Oct 30 22:14:14.412245 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.412209    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca" pod "bafbe951-ceb8-4e7b-b2c3-3b03647b6208" (UID: "bafbe951-ceb8-4e7b-b2c3-3b03647b6208"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:14.415330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.415289    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls" pod "bafbe951-ceb8-4e7b-b2c3-3b03647b6208" (UID: "bafbe951-ceb8-4e7b-b2c3-3b03647b6208"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:14.415453 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.415425    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x" pod "bafbe951-ceb8-4e7b-b2c3-3b03647b6208" (UID: "bafbe951-ceb8-4e7b-b2c3-3b03647b6208"). InnerVolumeSpecName "kube-api-access-xj88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:14.434118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.433899    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs" pod "bafbe951-ceb8-4e7b-b2c3-3b03647b6208" (UID: "bafbe951-ceb8-4e7b-b2c3-3b03647b6208"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:14.514361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.514279    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj88x\" (UniqueName: \"kubernetes.io/projected/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-kube-api-access-xj88x\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:14.514578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.514550    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:14.514728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.514583    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:14.514728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.514598    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bafbe951-ceb8-4e7b-b2c3-3b03647b6208-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:14.636211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.636162    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:14:14.636491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:14.636440    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" podUID="8a01db29-484f-46cc-9ef1-3ee984d81069" containerName="maas-api" containerID="cri-o://40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf" gracePeriod=30
Oct 30 22:14:15.169364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.169041    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:14:15.302692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.269608    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\") pod \"8a01db29-484f-46cc-9ef1-3ee984d81069\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") "
Oct 30 22:14:15.302692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.269693    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\") pod \"8a01db29-484f-46cc-9ef1-3ee984d81069\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") "
Oct 30 22:14:15.302692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.269910    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\") pod \"8a01db29-484f-46cc-9ef1-3ee984d81069\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") "
Oct 30 22:14:15.302692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.269978    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\" (UniqueName: \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\") pod \"8a01db29-484f-46cc-9ef1-3ee984d81069\" (UID: \"8a01db29-484f-46cc-9ef1-3ee984d81069\") "
Oct 30 22:14:15.302692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.278875    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca" pod "8a01db29-484f-46cc-9ef1-3ee984d81069" (UID: "8a01db29-484f-46cc-9ef1-3ee984d81069"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:15.397733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.397634    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a01db29-484f-46cc-9ef1-3ee984d81069-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.509087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.487375    4295 generic.go:356] "Generic (PLEG): container finished" podID="8a01db29-484f-46cc-9ef1-3ee984d81069" containerID="40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf" exitCode=0
Oct 30 22:14:15.509087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.487546    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" event={"ID":"8a01db29-484f-46cc-9ef1-3ee984d81069","Type":"ContainerDied","Data":"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"}
Oct 30 22:14:15.509087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.487642    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt" event={"ID":"8a01db29-484f-46cc-9ef1-3ee984d81069","Type":"ContainerDied","Data":"c8efd446c49a2fef633b01450d35d34fafb6149f63a4aafd0476e6146d8f6d18"}
Oct 30 22:14:15.509087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.487718    4295 scope.go:122] "RemoveContainer" containerID="40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"
Oct 30 22:14:15.509087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.487937    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"
Oct 30 22:14:15.566929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.565532    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt" pod "8a01db29-484f-46cc-9ef1-3ee984d81069" (UID: "8a01db29-484f-46cc-9ef1-3ee984d81069"). InnerVolumeSpecName "kube-api-access-tlckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:15.592436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.568907    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs" pod "8a01db29-484f-46cc-9ef1-3ee984d81069" (UID: "8a01db29-484f-46cc-9ef1-3ee984d81069"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:15.592436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.568978    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls" pod "8a01db29-484f-46cc-9ef1-3ee984d81069" (UID: "8a01db29-484f-46cc-9ef1-3ee984d81069"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:15.616215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.616186    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"
Oct 30 22:14:15.625758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.625715    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5" event={"ID":"bafbe951-ceb8-4e7b-b2c3-3b03647b6208","Type":"ContainerDied","Data":"c43dd60959875e9173cb759585a9a3ba78af5206430d1aa1d44e6f1fccd63cd5"}
Oct 30 22:14:15.780618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.779620    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlckt\" (UniqueName: \"kubernetes.io/projected/8a01db29-484f-46cc-9ef1-3ee984d81069-kube-api-access-tlckt\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.780618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.779679    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.780618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.779696    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a01db29-484f-46cc-9ef1-3ee984d81069-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.860377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.860302    4295 scope.go:122] "RemoveContainer" containerID="40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"
Oct 30 22:14:15.867346 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:15.867227    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf\": container with ID starting with 40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf not found: ID does not exist" containerID="40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"
Oct 30 22:14:15.867346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.867293    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf"} err="failed to get container status \"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf\": rpc error: code = NotFound desc = could not find container \"40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf\": container with ID starting with 40d8ba39a8cec774a22165bc623f30ae55042dbb162acdf304dd9f8b498065cf not found: ID does not exist"
Oct 30 22:14:15.867346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.867321    4295 scope.go:122] "RemoveContainer" containerID="061952593ece01d168dfdaca0a3eb0d1179a72068eab6b37d624c44855d85d87"
Oct 30 22:14:15.992079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:15.991996    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:14:16.028831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.006111    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4cce86f1-b4f6c584-lwgh5"]
Oct 30 22:14:16.104807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.104742    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:14:16.245769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.245583    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08235cc9-f49fcf47d-zffbt"]
Oct 30 22:14:16.706689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.699390    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:16.721433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.719780    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:16.805644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.804773    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:16.952580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:16.952480    4295 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:18.088720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:18.081152    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a01db29-484f-46cc-9ef1-3ee984d81069" path="/var/lib/kubelet/pods/8a01db29-484f-46cc-9ef1-3ee984d81069/volumes"
Oct 30 22:14:18.096017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:18.095982    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" path="/var/lib/kubelet/pods/bafbe951-ceb8-4e7b-b2c3-3b03647b6208/volumes"
Oct 30 22:14:20.184700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.184654    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:14:20.185410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185393    4295 cpu_manager.go:414] "containerMap: removing container" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" containerName="payload-pre-processing"
Oct 30 22:14:20.185469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185413    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" containerName="payload-pre-processing"
Oct 30 22:14:20.185469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185445    4295 cpu_manager.go:414] "containerMap: removing container" podUID="4bcc81dd-b855-43cb-afdc-690f7e931979" containerName="payload-processing"
Oct 30 22:14:20.185469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185450    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcc81dd-b855-43cb-afdc-690f7e931979" containerName="payload-processing"
Oct 30 22:14:20.185469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185459    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" containerName="maas-api"
Oct 30 22:14:20.185469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185464    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" containerName="maas-api"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185480    4295 cpu_manager.go:414] "containerMap: removing container" podUID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" containerName="payload-pre-processing"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185486    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" containerName="payload-pre-processing"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185493    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8a01db29-484f-46cc-9ef1-3ee984d81069" containerName="maas-api"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185497    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a01db29-484f-46cc-9ef1-3ee984d81069" containerName="maas-api"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185600    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcc81dd-b855-43cb-afdc-690f7e931979" containerName="payload-processing"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185610    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafbe951-ceb8-4e7b-b2c3-3b03647b6208" containerName="maas-api"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185618    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfaf1899-04bf-433d-9c2b-61b7575e7d4c" containerName="payload-pre-processing"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185624    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a01db29-484f-46cc-9ef1-3ee984d81069" containerName="maas-api"
Oct 30 22:14:20.185629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.185631    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="988d0b92-7666-4e98-ba42-7381649a0db5" containerName="payload-pre-processing"
Oct 30 22:14:20.189428 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.189407    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.260461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.260429    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:14:20.298716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.298653    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-781ccf27-openshift-default-dockercfg-ntd8j\""
Oct 30 22:14:20.301741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.301703    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.301893 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.301873    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zks\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.301967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.301920    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.301987    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302119 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.302099    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.302196    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.302229    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302391 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.302371    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.302427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.302417    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.406704    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.406764    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.406890    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.406922    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.407012    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.407127    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zks\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.407170    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.407230    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.407289    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.408189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.408477    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.411682 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.410380    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.412876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.412853    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.413086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.412877    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.440735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.415824    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.440735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.415869    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.493750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.490606    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zks\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.504697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.495516    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\") pod \"e2e-derive-781ccf27-openshift-default-79c47976f-24n2d\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") " pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:20.504697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:20.503092    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:21.012248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.012206    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:14:21.045928 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:14:21.045898    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544e6d2e_ba0f_4029_b7c8_a45fe75e2770.slice/crio-a22a185242543e7d42be8459fbf3a3a0d5ada4c8e5a1ea75d8157c0da8a78fb2 WatchSource:0}: Error finding container a22a185242543e7d42be8459fbf3a3a0d5ada4c8e5a1ea75d8157c0da8a78fb2: Status 404 returned error can't find the container with id a22a185242543e7d42be8459fbf3a3a0d5ada4c8e5a1ea75d8157c0da8a78fb2
Oct 30 22:14:21.048415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.048380    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:21.048481 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.048452    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:21.048515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.048484    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:21.872324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.872279    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerStarted","Data":"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"}
Oct 30 22:14:21.872754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:21.872385    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerStarted","Data":"a22a185242543e7d42be8459fbf3a3a0d5ada4c8e5a1ea75d8157c0da8a78fb2"}
Oct 30 22:14:22.058403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.058351    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podStartSLOduration=2.058336827 podStartE2EDuration="2.058336827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:22.056593725 +0000 UTC m=+1464.699187507" watchObservedRunningTime="2026-10-30 22:14:22.058336827 +0000 UTC m=+1464.700930697"
Oct 30 22:14:22.504201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.504170    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:22.510070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.510046    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:22.879649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.879578    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:22.880465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.880446    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:14:22.987500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.987471    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:14:22.987701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:22.987642    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" podUID="fcf1a254-d564-40d0-b6f5-1edac676ed33" containerName="maas-api" containerID="cri-o://4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364" gracePeriod=30
Oct 30 22:14:23.431503 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.431153    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:14:23.527769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.518016    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\") pod \"fcf1a254-d564-40d0-b6f5-1edac676ed33\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") "
Oct 30 22:14:23.528072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.528044    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\" (UniqueName: \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\") pod \"fcf1a254-d564-40d0-b6f5-1edac676ed33\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") "
Oct 30 22:14:23.528421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.528401    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\") pod \"fcf1a254-d564-40d0-b6f5-1edac676ed33\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") "
Oct 30 22:14:23.528632 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.528617    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\") pod \"fcf1a254-d564-40d0-b6f5-1edac676ed33\" (UID: \"fcf1a254-d564-40d0-b6f5-1edac676ed33\") "
Oct 30 22:14:23.532795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.525471    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls" pod "fcf1a254-d564-40d0-b6f5-1edac676ed33" (UID: "fcf1a254-d564-40d0-b6f5-1edac676ed33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:23.548983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.548916    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca" pod "fcf1a254-d564-40d0-b6f5-1edac676ed33" (UID: "fcf1a254-d564-40d0-b6f5-1edac676ed33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:23.551420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.551386    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs" pod "fcf1a254-d564-40d0-b6f5-1edac676ed33" (UID: "fcf1a254-d564-40d0-b6f5-1edac676ed33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:23.559704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.556704    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl" pod "fcf1a254-d564-40d0-b6f5-1edac676ed33" (UID: "fcf1a254-d564-40d0-b6f5-1edac676ed33"). InnerVolumeSpecName "kube-api-access-gnccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:23.645109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.645008    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1a254-d564-40d0-b6f5-1edac676ed33-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.645109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.645048    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.645109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.645065    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf1a254-d564-40d0-b6f5-1edac676ed33-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.645109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.645080    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnccl\" (UniqueName: \"kubernetes.io/projected/fcf1a254-d564-40d0-b6f5-1edac676ed33-kube-api-access-gnccl\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.718219 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:23.717653    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:14:23.887442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.887405    4295 generic.go:356] "Generic (PLEG): container finished" podID="fcf1a254-d564-40d0-b6f5-1edac676ed33" containerID="4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364" exitCode=0
Oct 30 22:14:23.887987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.887457    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"
Oct 30 22:14:23.887987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.887488    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" event={"ID":"fcf1a254-d564-40d0-b6f5-1edac676ed33","Type":"ContainerDied","Data":"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"}
Oct 30 22:14:23.887987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.887549    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt" event={"ID":"fcf1a254-d564-40d0-b6f5-1edac676ed33","Type":"ContainerDied","Data":"428cb760a695817ee5a3f0a15b8d487c1098026e6a6af14cd2e1f6cce341985a"}
Oct 30 22:14:23.887987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.887579    4295 scope.go:122] "RemoveContainer" containerID="4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"
Oct 30 22:14:23.898217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.898200    4295 scope.go:122] "RemoveContainer" containerID="4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"
Oct 30 22:14:23.898527 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:23.898487    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364\": container with ID starting with 4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364 not found: ID does not exist" containerID="4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"
Oct 30 22:14:23.898621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:23.898521    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364"} err="failed to get container status \"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364\": rpc error: code = NotFound desc = could not find container \"4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364\": container with ID starting with 4bb41b7127a2222c1f15aff5cd5cc64effb46b4dc3a381bd47d973e2e4fb5364 not found: ID does not exist"
Oct 30 22:14:24.073687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:24.066208    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:14:24.121087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:24.121040    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-643a82de-66d65fcb74-24twt"]
Oct 30 22:14:25.494945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:25.494913    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf1a254-d564-40d0-b6f5-1edac676ed33" path="/var/lib/kubelet/pods/fcf1a254-d564-40d0-b6f5-1edac676ed33/volumes"
Oct 30 22:14:26.153683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:26.153614    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:14:26.154258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:26.154174    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" containerID="cri-o://19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448" gracePeriod=30
Oct 30 22:14:27.682783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:27.682750    4295 patch_prober.go:29] interesting pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:27.683199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:27.682813    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:30.759744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.759636    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:14:30.761405 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.761376    4295 cpu_manager.go:414] "containerMap: removing container" podUID="fcf1a254-d564-40d0-b6f5-1edac676ed33" containerName="maas-api"
Oct 30 22:14:30.761573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.761560    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf1a254-d564-40d0-b6f5-1edac676ed33" containerName="maas-api"
Oct 30 22:14:30.761951 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.761930    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf1a254-d564-40d0-b6f5-1edac676ed33" containerName="maas-api"
Oct 30 22:14:30.830815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.830789    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:30.833348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.833322    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:14:30.941636 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.941608    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-781ccf27\""
Oct 30 22:14:30.952745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.951000    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxm5d\" (UniqueName: \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:30.952745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.951162    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:30.982455 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:30.982429    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-781ccf27-dockercfg-g6x6k\""
Oct 30 22:14:31.052516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.052429    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxm5d\" (UniqueName: \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.052708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.052532    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.053234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.053215    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.116223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.116196    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxm5d\" (UniqueName: \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\") pod \"payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.254561 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.254536    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.485537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.485515    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:14:31.559638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559612    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.559804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559766    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.559804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559787    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.559923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559803    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.559923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559846    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.559923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559884    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.560079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.559967    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.560079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560050    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.560182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560119    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\") pod \"de9111e2-a23a-4a6e-813f-57113be2996f\" (UID: \"de9111e2-a23a-4a6e-813f-57113be2996f\") "
Oct 30 22:14:31.560182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560145    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:31.560182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560142    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:31.560182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560160    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:31.560403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560374    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:31.560760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560735    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:31.560862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560806    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.560862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560828    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.560862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560843    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.560862 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.560857    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.561997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.561976    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:31.562231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.562218    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "kube-api-access-gz4l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:31.562484 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.562470    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:31.562656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.562644    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token" pod "de9111e2-a23a-4a6e-813f-57113be2996f" (UID: "de9111e2-a23a-4a6e-813f-57113be2996f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:31.639711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.639681    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:14:31.641088 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:14:31.640953    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c9bd0f_03c6_48fd_9916_cedc44583245.slice/crio-a8b43bcbe985f953d23b30a8cf8be185bf5126f15814f88b6f505c90e9c18913 WatchSource:0}: Error finding container a8b43bcbe985f953d23b30a8cf8be185bf5126f15814f88b6f505c90e9c18913: Status 404 returned error can't find the container with id a8b43bcbe985f953d23b30a8cf8be185bf5126f15814f88b6f505c90e9c18913
Oct 30 22:14:31.663201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.663176    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4l8\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-kube-api-access-gz4l8\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.663429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.663205    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9111e2-a23a-4a6e-813f-57113be2996f-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.663429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.663222    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9111e2-a23a-4a6e-813f-57113be2996f-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.663429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.663236    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9111e2-a23a-4a6e-813f-57113be2996f-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.663429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.663249    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9111e2-a23a-4a6e-813f-57113be2996f-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.972062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.972028    4295 generic.go:356] "Generic (PLEG): container finished" podID="de9111e2-a23a-4a6e-813f-57113be2996f" containerID="19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448" exitCode=0
Oct 30 22:14:31.972529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.972083    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"
Oct 30 22:14:31.972529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.972107    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerDied","Data":"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"}
Oct 30 22:14:31.972529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.972160    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h" event={"ID":"de9111e2-a23a-4a6e-813f-57113be2996f","Type":"ContainerDied","Data":"6f5c3a7f74e5769718f181f03a1c6d3fd5e701f5f91bc753df5f50f5714b4acd"}
Oct 30 22:14:31.972529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.972187    4295 scope.go:122] "RemoveContainer" containerID="19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"
Oct 30 22:14:31.973829 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.973796    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" event={"ID":"90c9bd0f-03c6-48fd-9916-cedc44583245","Type":"ContainerStarted","Data":"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"}
Oct 30 22:14:31.973936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.973845    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" event={"ID":"90c9bd0f-03c6-48fd-9916-cedc44583245","Type":"ContainerStarted","Data":"a8b43bcbe985f953d23b30a8cf8be185bf5126f15814f88b6f505c90e9c18913"}
Oct 30 22:14:31.973936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.973876    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:31.983008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.982995    4295 scope.go:122] "RemoveContainer" containerID="091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"
Oct 30 22:14:31.991609 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.991595    4295 scope.go:122] "RemoveContainer" containerID="19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"
Oct 30 22:14:31.991875 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:31.991857    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448\": container with ID starting with 19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448 not found: ID does not exist" containerID="19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"
Oct 30 22:14:31.991941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.991886    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448"} err="failed to get container status \"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448\": rpc error: code = NotFound desc = could not find container \"19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448\": container with ID starting with 19b1af3a94b9d631b6ea0fdace9a21974f0d8dd07da9f97acb74dff4fbbe3448 not found: ID does not exist"
Oct 30 22:14:31.991941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.991910    4295 scope.go:122] "RemoveContainer" containerID="091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"
Oct 30 22:14:31.992177 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:31.992156    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d\": container with ID starting with 091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d not found: ID does not exist" containerID="091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"
Oct 30 22:14:31.992224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:31.992184    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d"} err="failed to get container status \"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d\": rpc error: code = NotFound desc = could not find container \"091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d\": container with ID starting with 091c2a57753ceb887c0c72c085f042ddb4134a92c104a5ceca7f5c87c6f7294d not found: ID does not exist"
Oct 30 22:14:32.153096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:32.153037    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" podStartSLOduration=2.153020061 podStartE2EDuration="2.153020061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:32.150183189 +0000 UTC m=+1474.792776983" watchObservedRunningTime="2026-10-30 22:14:32.153020061 +0000 UTC m=+1474.795613851"
Oct 30 22:14:32.325792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:32.323686    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:14:32.325792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:32.323752    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h"]
Oct 30 22:14:33.494747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:33.494709    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" path="/var/lib/kubelet/pods/de9111e2-a23a-4a6e-813f-57113be2996f/volumes"
Oct 30 22:14:42.985259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:42.985187    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:14:49.658905 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.658868    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 22:14:49.659362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.659118    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy" containerID="cri-o://e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7" gracePeriod=30
Oct 30 22:14:49.708817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.708787    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:14:49.709024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.708997    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" containerID="cri-o://edf61fc359a0f2beae2311fc0a9149d8f4802ef74148e04751f680f1a053cba4" gracePeriod=30
Oct 30 22:14:49.712053 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.712027    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:14:49.712253 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.712232    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy" containerID="cri-o://cb3cd638dedff9f8f2d9161b7e1ec21c6a73f9d5ea13ef1cb29411b684c68de4" gracePeriod=30
Oct 30 22:14:49.759215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.759182    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:14:49.759418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:49.759396    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" containerID="cri-o://3233b8cb4bdeff4df00030523886697ad7a9384a6da21eb2589f49f893c57ed6" gracePeriod=30
Oct 30 22:14:52.910422 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:52.910385    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:52.910830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:52.910450    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:53.608945 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:53.608909    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:14:54.202072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:54.202034    4295 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:54.202426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:54.202101    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:55.008895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.008864    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:14:55.071032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.070992    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071069    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071130    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071233    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071323    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071404    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071475    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071544    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.071709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.071638    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\") pod \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\" (UID: \"f86b46f8-0826-42d7-ad00-ba0578ba6a2d\") "
Oct 30 22:14:55.073014 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.072982    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:55.073189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.073164    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.073401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.073374    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.073722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.073695    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.073819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.073764    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.079175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.079146    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "kube-api-access-hfj6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.079566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.079532    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:55.079865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.079635    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.081115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.081092    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token" pod "f86b46f8-0826-42d7-ad00-ba0578ba6a2d" (UID: "f86b46f8-0826-42d7-ad00-ba0578ba6a2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.123418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.123384    4295 generic.go:356] "Generic (PLEG): container finished" podID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerID="edf61fc359a0f2beae2311fc0a9149d8f4802ef74148e04751f680f1a053cba4" exitCode=0
Oct 30 22:14:55.123562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.123491    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" event={"ID":"bb54e478-285c-480e-a2aa-7bd1f3df90d2","Type":"ContainerDied","Data":"edf61fc359a0f2beae2311fc0a9149d8f4802ef74148e04751f680f1a053cba4"}
Oct 30 22:14:55.125204 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.125181    4295 generic.go:356] "Generic (PLEG): container finished" podID="64060740-fbde-4041-ad6d-a8311b91c08b" containerID="3233b8cb4bdeff4df00030523886697ad7a9384a6da21eb2589f49f893c57ed6" exitCode=0
Oct 30 22:14:55.125320 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.125258    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" event={"ID":"64060740-fbde-4041-ad6d-a8311b91c08b","Type":"ContainerDied","Data":"3233b8cb4bdeff4df00030523886697ad7a9384a6da21eb2589f49f893c57ed6"}
Oct 30 22:14:55.127092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.127067    4295 generic.go:356] "Generic (PLEG): container finished" podID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerID="cb3cd638dedff9f8f2d9161b7e1ec21c6a73f9d5ea13ef1cb29411b684c68de4" exitCode=0
Oct 30 22:14:55.127211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.127127    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" event={"ID":"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc","Type":"ContainerDied","Data":"cb3cd638dedff9f8f2d9161b7e1ec21c6a73f9d5ea13ef1cb29411b684c68de4"}
Oct 30 22:14:55.129078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.129054    4295 generic.go:356] "Generic (PLEG): container finished" podID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerID="e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7" exitCode=0
Oct 30 22:14:55.129185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.129098    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerDied","Data":"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"}
Oct 30 22:14:55.129185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.129136    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps" event={"ID":"f86b46f8-0826-42d7-ad00-ba0578ba6a2d","Type":"ContainerDied","Data":"884956e2fc192f9db8a329f22d707c2b7bc37c58e6ffb85ef391e405f1c3f7cc"}
Oct 30 22:14:55.129185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.129158    4295 scope.go:122] "RemoveContainer" containerID="e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"
Oct 30 22:14:55.129310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.129271    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"
Oct 30 22:14:55.133066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.133030    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:14:55.144361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.144340    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:14:55.145266 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.145248    4295 scope.go:122] "RemoveContainer" containerID="4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"
Oct 30 22:14:55.148911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.148892    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:14:55.154568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.154551    4295 scope.go:122] "RemoveContainer" containerID="e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"
Oct 30 22:14:55.154869 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:55.154849    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7\": container with ID starting with e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7 not found: ID does not exist" containerID="e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"
Oct 30 22:14:55.154944 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.154883    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7"} err="failed to get container status \"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7\": rpc error: code = NotFound desc = could not find container \"e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7\": container with ID starting with e6e3a76ae096d9f827adc51cb690ef4529bb4abb73fd836af8901d0ab88648a7 not found: ID does not exist"
Oct 30 22:14:55.154944 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.154906    4295 scope.go:122] "RemoveContainer" containerID="4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"
Oct 30 22:14:55.155186 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:14:55.155169    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2\": container with ID starting with 4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2 not found: ID does not exist" containerID="4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"
Oct 30 22:14:55.155242 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.155192    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2"} err="failed to get container status \"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2\": rpc error: code = NotFound desc = could not find container \"4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2\": container with ID starting with 4f0b0f921c6b4f3437879a2c11a22dbdbf5f0732dbf09924cb09a48c940e8bd2 not found: ID does not exist"
Oct 30 22:14:55.173033 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173006    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173056    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173094    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173126    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173193    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173299    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173340    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173366    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.173426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173407    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:55.173426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173410    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.173585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173566    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.173621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173578    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.173654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173622    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.173707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.173654    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\") pod \"64060740-fbde-4041-ad6d-a8311b91c08b\" (UID: \"64060740-fbde-4041-ad6d-a8311b91c08b\") "
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174227    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174256    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64060740-fbde-4041-ad6d-a8311b91c08b-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174268    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174276    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174286    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174294    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174302    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174310    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174318    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174346    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174355    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174365    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174374    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.174899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.174382    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfj6k\" (UniqueName: \"kubernetes.io/projected/f86b46f8-0826-42d7-ad00-ba0578ba6a2d-kube-api-access-hfj6k\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.175598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.175575    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.175755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.175739    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.176165 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.176148    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "kube-api-access-48tvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.176896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.176877    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo" pod "64060740-fbde-4041-ad6d-a8311b91c08b" (UID: "64060740-fbde-4041-ad6d-a8311b91c08b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:55.275409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275319    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275455    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275488    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275547    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275581    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275610    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275690    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275728    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275795    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275868    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275921    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.275966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275958    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.275984    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276040    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276076    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276135    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276160    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276242    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\") pod \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\" (UID: \"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc\") "
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276554    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.276810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276487    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\") pod \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\" (UID: \"bb54e478-285c-480e-a2aa-7bd1f3df90d2\") "
Oct 30 22:14:55.277207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.276951    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:55.277207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277020    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.277314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277268    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.277372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277329    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.277506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277484    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:55.277651 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277632    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277673    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48tvb\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-kube-api-access-48tvb\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277692    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277707    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277722    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277736    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.277760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277751    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64060740-fbde-4041-ad6d-a8311b91c08b-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.278049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277766    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.278049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277778    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64060740-fbde-4041-ad6d-a8311b91c08b-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.278049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277790    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64060740-fbde-4041-ad6d-a8311b91c08b-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.278049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.277804    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.278299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278256    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.278395 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278324    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.278395 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278384    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:55.278593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278567    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "kube-api-access-6kwh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.278756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278629    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.278850 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.278800    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.279322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.279117    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:55.279322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.279231    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.279509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.279481    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy" pod "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" (UID: "c7eb4f10-5947-4a49-832f-4eaf57e3f7dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:55.280265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.280245    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "kube-api-access-j9qlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.280334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.280319    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token" pod "bb54e478-285c-480e-a2aa-7bd1f3df90d2" (UID: "bb54e478-285c-480e-a2aa-7bd1f3df90d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:55.315479 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.315445    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 22:14:55.318558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.318537    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps"]
Oct 30 22:14:55.379420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379385    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379415    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qlh\" (UniqueName: \"kubernetes.io/projected/bb54e478-285c-480e-a2aa-7bd1f3df90d2-kube-api-access-j9qlh\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379428    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379440    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379448    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379456    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379464    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb54e478-285c-480e-a2aa-7bd1f3df90d2-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379472    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwh2\" (UniqueName: \"kubernetes.io/projected/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-kube-api-access-6kwh2\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379480    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379488    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.379610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.379503    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb54e478-285c-480e-a2aa-7bd1f3df90d2-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.494556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:55.494526    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" path="/var/lib/kubelet/pods/f86b46f8-0826-42d7-ad00-ba0578ba6a2d/volumes"
Oct 30 22:14:56.136586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.136503    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"
Oct 30 22:14:56.136586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.136522    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv" event={"ID":"bb54e478-285c-480e-a2aa-7bd1f3df90d2","Type":"ContainerDied","Data":"2099a2dd41fbcf000163e6a583f282b589d6a562fea3d01cc67bc52c83ba4c2e"}
Oct 30 22:14:56.136586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.136581    4295 scope.go:122] "RemoveContainer" containerID="edf61fc359a0f2beae2311fc0a9149d8f4802ef74148e04751f680f1a053cba4"
Oct 30 22:14:56.138369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.138341    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784" event={"ID":"64060740-fbde-4041-ad6d-a8311b91c08b","Type":"ContainerDied","Data":"0db3bbce71e4c1d7b47541d8c6534c102268fd13533ecc86e94df68925ce1cd8"}
Oct 30 22:14:56.138477 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.138378    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"
Oct 30 22:14:56.139981 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.139963    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"
Oct 30 22:14:56.140073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.139996    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px" event={"ID":"c7eb4f10-5947-4a49-832f-4eaf57e3f7dc","Type":"ContainerDied","Data":"da1399ed1acd36b2e685d3729c2ad6333d9e15fb9d47c490e00eabed2ece37b3"}
Oct 30 22:14:56.147573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.147554    4295 scope.go:122] "RemoveContainer" containerID="3233b8cb4bdeff4df00030523886697ad7a9384a6da21eb2589f49f893c57ed6"
Oct 30 22:14:56.157035 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.157017    4295 scope.go:122] "RemoveContainer" containerID="cb3cd638dedff9f8f2d9161b7e1ec21c6a73f9d5ea13ef1cb29411b684c68de4"
Oct 30 22:14:56.317309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.317272    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:14:56.367976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.367940    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px"]
Oct 30 22:14:56.445502 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.445468    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:14:56.455026 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.454998    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv"]
Oct 30 22:14:56.570173 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.570143    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:14:56.585854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:56.585816    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784"]
Oct 30 22:14:57.495061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:57.495029    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" path="/var/lib/kubelet/pods/64060740-fbde-4041-ad6d-a8311b91c08b/volumes"
Oct 30 22:14:57.495592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:57.495577    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" path="/var/lib/kubelet/pods/bb54e478-285c-480e-a2aa-7bd1f3df90d2/volumes"
Oct 30 22:14:57.496110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:14:57.496096    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" path="/var/lib/kubelet/pods/c7eb4f10-5947-4a49-832f-4eaf57e3f7dc/volumes"
Oct 30 22:15:00.194138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194100    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:00.194696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194676    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.194696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194694    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194703    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194709    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194719    4295 cpu_manager.go:414] "containerMap: removing container" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194724    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194750    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194755    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194762    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194767    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194779    4295 cpu_manager.go:414] "containerMap: removing container" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.194789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194783    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194794    4295 cpu_manager.go:414] "containerMap: removing container" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194799    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194881    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb54e478-285c-480e-a2aa-7bd1f3df90d2" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194888    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eb4f10-5947-4a49-832f-4eaf57e3f7dc" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194897    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194903    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9111e2-a23a-4a6e-813f-57113be2996f" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194909    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194914    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86b46f8-0826-42d7-ad00-ba0578ba6a2d" containerName="istio-proxy"
Oct 30 22:15:00.195052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.194919    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="64060740-fbde-4041-ad6d-a8311b91c08b" containerName="istio-proxy"
Oct 30 22:15:00.200222 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.200197    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.235125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.235096    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:00.239017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.239000    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.253212 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.253189    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:00.256949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.256933    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.270966 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.270945    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.271172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.271157    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2v2x\" (UniqueName: \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.282443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.282425    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:15:00.305910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.305884    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:00.346750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.346715    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:00.351517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.351490    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:00.373041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.372999    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2v2x\" (UniqueName: \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.373652 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.373620    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.373807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.373724    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkt8b\" (UniqueName: \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.373807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.373773    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.373986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.373817    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.373986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.373854    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bmd\" (UniqueName: \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.374619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.374597    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.442567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.442541    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2v2x\" (UniqueName: \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.475218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475126    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.475373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475217    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkt8b\" (UniqueName: \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.475373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475262    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.475373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475310    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bmd\" (UniqueName: \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.475965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475940    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.476023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.475981    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.513595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.513567    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:00.542726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.542702    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bmd\" (UniqueName: \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\") pod \"maas-api-key-cleanup-29889975-frkc7\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.542937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.542914    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkt8b\" (UniqueName: \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.552700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.552674    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:00.573152 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.572942    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:00.900961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:00.900935    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:00.902591 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:00.902561    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbb7d11_26b6_4785_8fb1_e133d7a64c85.slice/crio-20db42dd6f07c70b27b8bcf405a04da5dc4abfafc5b2a65cbb3d5e4d0db4c1ed WatchSource:0}: Error finding container 20db42dd6f07c70b27b8bcf405a04da5dc4abfafc5b2a65cbb3d5e4d0db4c1ed: Status 404 returned error can't find the container with id 20db42dd6f07c70b27b8bcf405a04da5dc4abfafc5b2a65cbb3d5e4d0db4c1ed
Oct 30 22:15:01.010267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:01.010234    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:01.019503 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:01.019476    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:01.021597 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:01.021575    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8532a14f_5a3b_4d42_b005_05a3fc1a98fe.slice/crio-91eabfa1200810d3634980897102968bfdd65c3deeb3073dda3e1cb59ab1508d WatchSource:0}: Error finding container 91eabfa1200810d3634980897102968bfdd65c3deeb3073dda3e1cb59ab1508d: Status 404 returned error can't find the container with id 91eabfa1200810d3634980897102968bfdd65c3deeb3073dda3e1cb59ab1508d
Oct 30 22:15:01.176152 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:01.176063    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerStarted","Data":"953a0716969ced57028538fd71ef78639edccd62a9dc8ad1b82422a613bb229c"}
Oct 30 22:15:01.177334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:01.177303    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" event={"ID":"3fbb7d11-26b6-4785-8fb1-e133d7a64c85","Type":"ContainerStarted","Data":"20db42dd6f07c70b27b8bcf405a04da5dc4abfafc5b2a65cbb3d5e4d0db4c1ed"}
Oct 30 22:15:01.178520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:01.178499    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" event={"ID":"8532a14f-5a3b-4d42-b005-05a3fc1a98fe","Type":"ContainerStarted","Data":"91eabfa1200810d3634980897102968bfdd65c3deeb3073dda3e1cb59ab1508d"}
Oct 30 22:15:02.199954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.199919    4295 generic.go:356] "Generic (PLEG): container finished" podID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerID="865a40253b843be48fa9861655c4c1746bde1bd8324a0f2e182df183748c1196" exitCode=60
Oct 30 22:15:02.200473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.200018    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" event={"ID":"8532a14f-5a3b-4d42-b005-05a3fc1a98fe","Type":"ContainerDied","Data":"865a40253b843be48fa9861655c4c1746bde1bd8324a0f2e182df183748c1196"}
Oct 30 22:15:02.200473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.200338    4295 scope.go:122] "RemoveContainer" containerID="865a40253b843be48fa9861655c4c1746bde1bd8324a0f2e182df183748c1196"
Oct 30 22:15:02.201977 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.201951    4295 generic.go:356] "Generic (PLEG): container finished" podID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerID="212d2b602e2c6e91d7dba60d71626a1dbd24e48f56c324d022d3ff6349b4fbaf" exitCode=60
Oct 30 22:15:02.202104 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.202020    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerDied","Data":"212d2b602e2c6e91d7dba60d71626a1dbd24e48f56c324d022d3ff6349b4fbaf"}
Oct 30 22:15:02.202300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.202281    4295 scope.go:122] "RemoveContainer" containerID="212d2b602e2c6e91d7dba60d71626a1dbd24e48f56c324d022d3ff6349b4fbaf"
Oct 30 22:15:02.204778 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.204752    4295 generic.go:356] "Generic (PLEG): container finished" podID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerID="d7364f2f939c4d65d0dfd07e0274cb11e869cc454beab51f47faa5cc28670e2d" exitCode=60
Oct 30 22:15:02.204874 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.204829    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" event={"ID":"3fbb7d11-26b6-4785-8fb1-e133d7a64c85","Type":"ContainerDied","Data":"d7364f2f939c4d65d0dfd07e0274cb11e869cc454beab51f47faa5cc28670e2d"}
Oct 30 22:15:02.205012 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:02.204998    4295 scope.go:122] "RemoveContainer" containerID="d7364f2f939c4d65d0dfd07e0274cb11e869cc454beab51f47faa5cc28670e2d"
Oct 30 22:15:03.217996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.217963    4295 generic.go:356] "Generic (PLEG): container finished" podID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerID="0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0" exitCode=60
Oct 30 22:15:03.218331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.218086    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" event={"ID":"8532a14f-5a3b-4d42-b005-05a3fc1a98fe","Type":"ContainerDied","Data":"0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0"}
Oct 30 22:15:03.218331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.218139    4295 scope.go:122] "RemoveContainer" containerID="865a40253b843be48fa9861655c4c1746bde1bd8324a0f2e182df183748c1196"
Oct 30 22:15:03.218421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.218393    4295 scope.go:122] "RemoveContainer" containerID="0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0"
Oct 30 22:15:03.218787 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:03.218763    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj_odh-ai-gateway-infra(8532a14f-5a3b-4d42-b005-05a3fc1a98fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe"
Oct 30 22:15:03.220070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.220051    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerStarted","Data":"01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"}
Oct 30 22:15:03.221923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.221897    4295 generic.go:356] "Generic (PLEG): container finished" podID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerID="8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f" exitCode=60
Oct 30 22:15:03.222029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.221941    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" event={"ID":"3fbb7d11-26b6-4785-8fb1-e133d7a64c85","Type":"ContainerDied","Data":"8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f"}
Oct 30 22:15:03.222217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.222201    4295 scope.go:122] "RemoveContainer" containerID="8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f"
Oct 30 22:15:03.222464 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:03.222445    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7_odh-ai-gateway-infra(3fbb7d11-26b6-4785-8fb1-e133d7a64c85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85"
Oct 30 22:15:03.389165 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.389145    4295 scope.go:122] "RemoveContainer" containerID="d7364f2f939c4d65d0dfd07e0274cb11e869cc454beab51f47faa5cc28670e2d"
Oct 30 22:15:03.485570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:03.485468    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" podStartSLOduration=3.485453004 podStartE2EDuration="3.485453004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.483733814 +0000 UTC m=+1506.126327603" watchObservedRunningTime="2026-10-30 22:15:03.485453004 +0000 UTC m=+1506.128046855"
Oct 30 22:15:04.229767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.229741    4295 scope.go:122] "RemoveContainer" containerID="0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0"
Oct 30 22:15:04.230203 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:04.230106    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj_odh-ai-gateway-infra(8532a14f-5a3b-4d42-b005-05a3fc1a98fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe"
Oct 30 22:15:04.231299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.231271    4295 generic.go:356] "Generic (PLEG): container finished" podID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerID="01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98" exitCode=60
Oct 30 22:15:04.231417 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.231351    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerDied","Data":"01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"}
Oct 30 22:15:04.231494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.231414    4295 scope.go:122] "RemoveContainer" containerID="212d2b602e2c6e91d7dba60d71626a1dbd24e48f56c324d022d3ff6349b4fbaf"
Oct 30 22:15:04.231595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.231576    4295 scope.go:122] "RemoveContainer" containerID="01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"
Oct 30 22:15:04.231964 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:04.231930    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-frkc7_odh-ai-gateway-infra(b1a41d0b-bea9-436e-b688-ba9ee39b3e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31"
Oct 30 22:15:04.233195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:04.233182    4295 scope.go:122] "RemoveContainer" containerID="8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f"
Oct 30 22:15:04.233436 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:04.233420    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7_odh-ai-gateway-infra(3fbb7d11-26b6-4785-8fb1-e133d7a64c85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85"
Oct 30 22:15:05.241320 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:05.241292    4295 scope.go:122] "RemoveContainer" containerID="01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"
Oct 30 22:15:05.241735 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:05.241695    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-frkc7_odh-ai-gateway-infra(b1a41d0b-bea9-436e-b688-ba9ee39b3e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31"
Oct 30 22:15:07.786178 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.786141    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:15:07.791409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.791386    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.890304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.890268    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.890515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.890385    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.890515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.890460    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.890515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.890478    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd7n\" (UniqueName: \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.898309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.898282    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ef8140e-serving-cert\""
Oct 30 22:15:07.901950 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.901926    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:15:07.992347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.992313    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.992535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.992396    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.992535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.992416    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd7n\" (UniqueName: \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.992535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.992525    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.992697 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:07.992680    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-serving-cert: secret "maas-api-e2e-ait-3ef8140e-serving-cert" not found
Oct 30 22:15:07.992776 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:07.992759    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls podName:15c68ef0-ef33-4195-b1c2-93c3e67107cf nodeName:}" failed. No retries permitted until 2026-10-30 22:15:08.492736198 +0000 UTC m=+1511.135329988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls") pod "maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" (UID: "15c68ef0-ef33-4195-b1c2-93c3e67107cf") : secret "maas-api-e2e-ait-3ef8140e-serving-cert" not found
Oct 30 22:15:07.993210 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.993193    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:07.994926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:07.994904    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:08.073061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:08.072965    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd7n\" (UniqueName: \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:08.508402 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:08.508358    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:08.510945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:08.510912    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") pod \"maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:08.706100 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:08.706058    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:09.109655 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:09.109622    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:15:09.110097 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:09.110067    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c68ef0_ef33_4195_b1c2_93c3e67107cf.slice/crio-3a91d0fd64e65e750e1f6b968695881a963936330c198bba8f8b4482edd82aec WatchSource:0}: Error finding container 3a91d0fd64e65e750e1f6b968695881a963936330c198bba8f8b4482edd82aec: Status 404 returned error can't find the container with id 3a91d0fd64e65e750e1f6b968695881a963936330c198bba8f8b4482edd82aec
Oct 30 22:15:09.296324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:09.296288    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" event={"ID":"15c68ef0-ef33-4195-b1c2-93c3e67107cf","Type":"ContainerStarted","Data":"3a91d0fd64e65e750e1f6b968695881a963936330c198bba8f8b4482edd82aec"}
Oct 30 22:15:10.359881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.347281    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" event={"ID":"15c68ef0-ef33-4195-b1c2-93c3e67107cf","Type":"ContainerStarted","Data":"c9b6da653c4db5f7450541ce731b448ebc56bc3a1ad518820da85f83c446fbe0"}
Oct 30 22:15:10.403725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.403646    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:10.536403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.536347    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" podStartSLOduration=2.670111312 podStartE2EDuration="3.536330844s" totalImagesPullingTime="866.219532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:10.534921474 +0000 UTC m=+1513.177515267" watchObservedRunningTime="2026-10-30 22:15:10.536330844 +0000 UTC m=+1513.178924633"
Oct 30 22:15:10.754650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.754598    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:15:10.762887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.762846    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.867836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.867792    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.867996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.867904    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.867996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.867953    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fphl\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868035    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868070    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868101    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868148    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868221 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868190    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868298    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.868374 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.868347    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:15:10.869569 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.869547    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7422b4b3-openshift-default-dockercfg-rgnmp\""
Oct 30 22:15:10.970821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.970783    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.971091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.971072    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fphl\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.971526 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.971504    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.972633 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.972614    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.973277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.973260    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.973757 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.973203    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.974062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.973632    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.979875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.979851    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.986816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.986784    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.987276 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.987254    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.987425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.987411    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.988098 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.988075    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.988205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.985983    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:10.988351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:10.988333    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.093452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.091573    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.100618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.100587    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.100995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.100967    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.102044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.102014    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fphl\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\") pod \"e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") " pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.248352 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.248314    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:11.721557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.721530    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:15:11.722006 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:11.721986    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfb937b_688b_4487_9507_7dc1c1e0ac7e.slice/crio-e64dcde7888b0793c56bc91ebff046011789815a8c63242ecb704c58ae9f339a WatchSource:0}: Error finding container e64dcde7888b0793c56bc91ebff046011789815a8c63242ecb704c58ae9f339a: Status 404 returned error can't find the container with id e64dcde7888b0793c56bc91ebff046011789815a8c63242ecb704c58ae9f339a
Oct 30 22:15:11.724860 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.724826    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:11.724951 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.724894    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:11.724951 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:11.724929    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:12.453072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:12.453036    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerStarted","Data":"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"}
Oct 30 22:15:12.453224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:12.453099    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerStarted","Data":"e64dcde7888b0793c56bc91ebff046011789815a8c63242ecb704c58ae9f339a"}
Oct 30 22:15:12.661362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:12.657791    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podStartSLOduration=2.6577643010000003 podStartE2EDuration="2.657764301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:12.650061498 +0000 UTC m=+1515.292655287" watchObservedRunningTime="2026-10-30 22:15:12.657764301 +0000 UTC m=+1515.300358084"
Oct 30 22:15:13.248497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.248464    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:13.253590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.253568    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:13.370202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.370168    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:15:13.374314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.374291    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.429581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.429537    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.429782 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.429717    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.429839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.429815    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn6d\" (UniqueName: \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.429994 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.429981    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.459448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.459419    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:13.460429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.460409    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:15:13.479067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.479043    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-781ccf27-serving-cert\""
Oct 30 22:15:13.486366 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.486342    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:15:13.531275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.531191    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn6d\" (UniqueName: \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.531424 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.531369    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.531508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.531460    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.531615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.531586    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.532436 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:13.532404    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-serving-cert: secret "maas-api-e2e-derive-781ccf27-serving-cert" not found
Oct 30 22:15:13.532555 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:13.532482    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls podName:df00b75d-4ed8-4ad2-abc4-da8f0a309f83 nodeName:}" failed. No retries permitted until 2026-10-30 22:15:14.032462645 +0000 UTC m=+1516.675056427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls") pod "maas-api-e2e-derive-781ccf27-5664496bfc-95r59" (UID: "df00b75d-4ed8-4ad2-abc4-da8f0a309f83") : secret "maas-api-e2e-derive-781ccf27-serving-cert" not found
Oct 30 22:15:13.532555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.532490    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.539131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.535917    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:13.595452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:13.595421    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn6d\" (UniqueName: \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:14.092160 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:14.092117    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:14.094889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:14.094868    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") pod \"maas-api-e2e-derive-781ccf27-5664496bfc-95r59\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:14.288954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:14.288916    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:14.486433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:14.486403    4295 scope.go:122] "RemoveContainer" containerID="01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"
Oct 30 22:15:14.677468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:14.677439    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:15:14.678840 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:14.678818    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf00b75d_4ed8_4ad2_abc4_da8f0a309f83.slice/crio-7ef9a909e46d38179406213f1efb851c2f963db2aa2203c94e7cb0449ccfd604 WatchSource:0}: Error finding container 7ef9a909e46d38179406213f1efb851c2f963db2aa2203c94e7cb0449ccfd604: Status 404 returned error can't find the container with id 7ef9a909e46d38179406213f1efb851c2f963db2aa2203c94e7cb0449ccfd604
Oct 30 22:15:15.473198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.473158    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" event={"ID":"df00b75d-4ed8-4ad2-abc4-da8f0a309f83","Type":"ContainerStarted","Data":"7ef9a909e46d38179406213f1efb851c2f963db2aa2203c94e7cb0449ccfd604"}
Oct 30 22:15:15.474995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.474970    4295 generic.go:356] "Generic (PLEG): container finished" podID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerID="67b0b4e149905880ebd411b4e882ffd49c87688165393815042ae91163438b55" exitCode=60
Oct 30 22:15:15.475102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.475044    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerDied","Data":"67b0b4e149905880ebd411b4e882ffd49c87688165393815042ae91163438b55"}
Oct 30 22:15:15.475102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.475097    4295 scope.go:122] "RemoveContainer" containerID="01a2bc84703c0f6084dcd758352697165382403ad488d311fa4fdedf150bda98"
Oct 30 22:15:15.475657 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.475637    4295 scope.go:122] "RemoveContainer" containerID="67b0b4e149905880ebd411b4e882ffd49c87688165393815042ae91163438b55"
Oct 30 22:15:15.476021 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:15.475997    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-frkc7_odh-ai-gateway-infra(b1a41d0b-bea9-436e-b688-ba9ee39b3e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31"
Oct 30 22:15:15.486379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:15.486350    4295 scope.go:122] "RemoveContainer" containerID="0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0"
Oct 30 22:15:16.486048 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.485994    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" event={"ID":"df00b75d-4ed8-4ad2-abc4-da8f0a309f83","Type":"ContainerStarted","Data":"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"}
Oct 30 22:15:16.486504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.486238    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:16.488535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.488511    4295 generic.go:356] "Generic (PLEG): container finished" podID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerID="ba990de18fc3e6b8951a037837b97c1642ca023566c7b5bba8fa388b148cd9eb" exitCode=60
Oct 30 22:15:16.488720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.488585    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" event={"ID":"8532a14f-5a3b-4d42-b005-05a3fc1a98fe","Type":"ContainerDied","Data":"ba990de18fc3e6b8951a037837b97c1642ca023566c7b5bba8fa388b148cd9eb"}
Oct 30 22:15:16.488720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.488644    4295 scope.go:122] "RemoveContainer" containerID="0e1ee4d78efff37c18e520170ab6409696faf3c00046ecfe04d18265b1febca0"
Oct 30 22:15:16.489602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.489581    4295 scope.go:122] "RemoveContainer" containerID="ba990de18fc3e6b8951a037837b97c1642ca023566c7b5bba8fa388b148cd9eb"
Oct 30 22:15:16.490114 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:16.489904    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj_odh-ai-gateway-infra(8532a14f-5a3b-4d42-b005-05a3fc1a98fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe"
Oct 30 22:15:16.609608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.609524    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:16.648325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.648267    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" podStartSLOduration=2.832789617 podStartE2EDuration="3.648248525s" totalImagesPullingTime="815.458908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:16.617768861 +0000 UTC m=+1519.260362658" watchObservedRunningTime="2026-10-30 22:15:16.648248525 +0000 UTC m=+1519.290842317"
Oct 30 22:15:16.835722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.833745    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:16.845688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.840273    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\") pod \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") "
Oct 30 22:15:16.845688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.840655    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\" (UniqueName: \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\") pod \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\" (UID: \"b1a41d0b-bea9-436e-b688-ba9ee39b3e31\") "
Oct 30 22:15:16.861735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.860013    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd" pod "b1a41d0b-bea9-436e-b688-ba9ee39b3e31" (UID: "b1a41d0b-bea9-436e-b688-ba9ee39b3e31"). InnerVolumeSpecName "kube-api-access-27bmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:16.861735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.860913    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca" pod "b1a41d0b-bea9-436e-b688-ba9ee39b3e31" (UID: "b1a41d0b-bea9-436e-b688-ba9ee39b3e31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:16.942594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.942554    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27bmd\" (UniqueName: \"kubernetes.io/projected/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-kube-api-access-27bmd\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.942594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:16.942590    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a41d0b-bea9-436e-b688-ba9ee39b3e31-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.458888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.458854    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:15:17.487021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.486995    4295 scope.go:122] "RemoveContainer" containerID="8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f"
Oct 30 22:15:17.498593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.498569    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"
Oct 30 22:15:17.498593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.498588    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7" event={"ID":"b1a41d0b-bea9-436e-b688-ba9ee39b3e31","Type":"ContainerDied","Data":"953a0716969ced57028538fd71ef78639edccd62a9dc8ad1b82422a613bb229c"}
Oct 30 22:15:17.498806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.498642    4295 scope.go:122] "RemoveContainer" containerID="67b0b4e149905880ebd411b4e882ffd49c87688165393815042ae91163438b55"
Oct 30 22:15:17.756987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.756953    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:17.759955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.759933    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-frkc7"]
Oct 30 22:15:17.890071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:17.890039    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:18.039075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.039054    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:18.064609 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.064578    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\") pod \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") "
Oct 30 22:15:18.064806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.064715    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\" (UniqueName: \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\") pod \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\" (UID: \"8532a14f-5a3b-4d42-b005-05a3fc1a98fe\") "
Oct 30 22:15:18.065040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.065017    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca" pod "8532a14f-5a3b-4d42-b005-05a3fc1a98fe" (UID: "8532a14f-5a3b-4d42-b005-05a3fc1a98fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.065357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.065340    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.066987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.066966    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b" pod "8532a14f-5a3b-4d42-b005-05a3fc1a98fe" (UID: "8532a14f-5a3b-4d42-b005-05a3fc1a98fe"). InnerVolumeSpecName "kube-api-access-vkt8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.167307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.167272    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkt8b\" (UniqueName: \"kubernetes.io/projected/8532a14f-5a3b-4d42-b005-05a3fc1a98fe-kube-api-access-vkt8b\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.506799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.506765    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"
Oct 30 22:15:18.507257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.506798    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj" event={"ID":"8532a14f-5a3b-4d42-b005-05a3fc1a98fe","Type":"ContainerDied","Data":"91eabfa1200810d3634980897102968bfdd65c3deeb3073dda3e1cb59ab1508d"}
Oct 30 22:15:18.507257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.506864    4295 scope.go:122] "RemoveContainer" containerID="ba990de18fc3e6b8951a037837b97c1642ca023566c7b5bba8fa388b148cd9eb"
Oct 30 22:15:18.510070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.510046    4295 generic.go:356] "Generic (PLEG): container finished" podID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerID="dc3a580e13fcdf1c95894918040200429b66d9a08971fec1c802e52c18bcc4bb" exitCode=60
Oct 30 22:15:18.510194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.510073    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" event={"ID":"3fbb7d11-26b6-4785-8fb1-e133d7a64c85","Type":"ContainerDied","Data":"dc3a580e13fcdf1c95894918040200429b66d9a08971fec1c802e52c18bcc4bb"}
Oct 30 22:15:18.510604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.510587    4295 scope.go:122] "RemoveContainer" containerID="dc3a580e13fcdf1c95894918040200429b66d9a08971fec1c802e52c18bcc4bb"
Oct 30 22:15:18.510999 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:18.510977    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7_odh-ai-gateway-infra(3fbb7d11-26b6-4785-8fb1-e133d7a64c85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85"
Oct 30 22:15:18.550200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.550180    4295 scope.go:122] "RemoveContainer" containerID="8a9e6f2bcee60a1a3142cd1827fcd06e9475eec87bc023aad6afe5dc0ad6785f"
Oct 30 22:15:18.690871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.690841    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:18.705212 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:18.705187    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889975-8dhzj"]
Oct 30 22:15:19.494276 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:19.494235    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" path="/var/lib/kubelet/pods/8532a14f-5a3b-4d42-b005-05a3fc1a98fe/volumes"
Oct 30 22:15:19.494798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:19.494777    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" path="/var/lib/kubelet/pods/b1a41d0b-bea9-436e-b688-ba9ee39b3e31/volumes"
Oct 30 22:15:19.971688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:19.970294    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.108867    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109776    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109796    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109830    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109840    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109853    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.109861    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110052    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110064    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110390    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110407    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110418    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110433    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.122835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.110443    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.213426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.212154    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:15:20.213426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.212300    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.344744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.344030    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7422b4b3\""
Oct 30 22:15:20.344744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.344445    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7422b4b3-dockercfg-dgmqd\""
Oct 30 22:15:20.389567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.389530    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:15:20.390428 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.390399    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.390428 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.390427    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.390610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.390462    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.390610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.390468    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532a14f-5a3b-4d42-b005-05a3fc1a98fe" containerName="cleanup"
Oct 30 22:15:20.390777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.390653    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a41d0b-bea9-436e-b688-ba9ee39b3e31" containerName="cleanup"
Oct 30 22:15:20.426357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.425181    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:15:20.426357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.425341    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.444596 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.444546    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwrv\" (UniqueName: \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.445054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.445031    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.482980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.482954    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:20.526725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.526700    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"
Oct 30 22:15:20.526881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.526695    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7" event={"ID":"3fbb7d11-26b6-4785-8fb1-e133d7a64c85","Type":"ContainerDied","Data":"20db42dd6f07c70b27b8bcf405a04da5dc4abfafc5b2a65cbb3d5e4d0db4c1ed"}
Oct 30 22:15:20.526881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.526839    4295 scope.go:122] "RemoveContainer" containerID="dc3a580e13fcdf1c95894918040200429b66d9a08971fec1c802e52c18bcc4bb"
Oct 30 22:15:20.546780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.546754    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.546866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.546830    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwrv\" (UniqueName: \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.546948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.546933    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztphx\" (UniqueName: \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.547007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.546982    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.547447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.547428    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.611040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.610962    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwrv\" (UniqueName: \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\") pod \"payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") " pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.647913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.647878    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\" (UniqueName: \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\") pod \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") "
Oct 30 22:15:20.648106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.647989    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\") pod \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\" (UID: \"3fbb7d11-26b6-4785-8fb1-e133d7a64c85\") "
Oct 30 22:15:20.648374 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.648345    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca" pod "3fbb7d11-26b6-4785-8fb1-e133d7a64c85" (UID: "3fbb7d11-26b6-4785-8fb1-e133d7a64c85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.648640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.648620    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztphx\" (UniqueName: \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.648750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.648707    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.648844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.648827    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.649451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.649424    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.650399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.650378    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x" pod "3fbb7d11-26b6-4785-8fb1-e133d7a64c85" (UID: "3fbb7d11-26b6-4785-8fb1-e133d7a64c85"). InnerVolumeSpecName "kube-api-access-w2v2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.712322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.712289    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztphx\" (UniqueName: \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\") pod \"payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.750113 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.750061    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2v2x\" (UniqueName: \"kubernetes.io/projected/3fbb7d11-26b6-4785-8fb1-e133d7a64c85-kube-api-access-w2v2x\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.768588 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.768558    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:20.781915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.781893    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:20.967784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.967748    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:20.970888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:20.970862    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce513-29889975-slkp7"]
Oct 30 22:15:21.169045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.168916    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:15:21.207101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.207070    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:15:21.207452 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:21.207423    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f873a6_2c04_4e13_a5e8_f5ff41fdb4a8.slice/crio-d7e739188f3d69e33749919c70ea4719d44d39ee5f92a6cb0af89e21723d74a7 WatchSource:0}: Error finding container d7e739188f3d69e33749919c70ea4719d44d39ee5f92a6cb0af89e21723d74a7: Status 404 returned error can't find the container with id d7e739188f3d69e33749919c70ea4719d44d39ee5f92a6cb0af89e21723d74a7
Oct 30 22:15:21.494137 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.494104    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" path="/var/lib/kubelet/pods/3fbb7d11-26b6-4785-8fb1-e133d7a64c85/volumes"
Oct 30 22:15:21.535746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.535708    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" event={"ID":"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8","Type":"ContainerStarted","Data":"10a00668241cfcf43488330b636dbb51a1c5977d0852e1ceff85eba2d368b6a3"}
Oct 30 22:15:21.535953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.535777    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:21.535953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.535803    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" event={"ID":"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8","Type":"ContainerStarted","Data":"d7e739188f3d69e33749919c70ea4719d44d39ee5f92a6cb0af89e21723d74a7"}
Oct 30 22:15:21.537358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.537330    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" event={"ID":"55c46161-3737-41c4-914e-58cb1682ba14","Type":"ContainerStarted","Data":"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"}
Oct 30 22:15:21.537486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.537367    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" event={"ID":"55c46161-3737-41c4-914e-58cb1682ba14","Type":"ContainerStarted","Data":"3c8bfca4f372c7c82c3b471e3316ea202583158127c737c7ea15a931f5e2c005"}
Oct 30 22:15:21.537486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.537408    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:21.659824 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.659754    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" podStartSLOduration=1.659739745 podStartE2EDuration="1.659739745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.657616663 +0000 UTC m=+1524.300210453" watchObservedRunningTime="2026-10-30 22:15:21.659739745 +0000 UTC m=+1524.302333567"
Oct 30 22:15:21.870226 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:21.867681    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" podStartSLOduration=1.867636981 podStartE2EDuration="1.867636981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.829982938 +0000 UTC m=+1524.472576728" watchObservedRunningTime="2026-10-30 22:15:21.867636981 +0000 UTC m=+1524.510230771"
Oct 30 22:15:22.503645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:22.503619    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:15:23.607332 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:23.607297    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:15:32.546886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:32.546859    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:15:32.547309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:32.546923    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:15:47.573688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.570222    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:15:47.597939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.586752    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.598393 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.598152    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.598570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.598556    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.610718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.598645    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.636017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.635984    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.636259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.636241    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.652697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.636582    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.652968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.652943    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.659759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.659726    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbb7d11-26b6-4785-8fb1-e133d7a64c85" containerName="cleanup"
Oct 30 22:15:47.693687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.684032    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:15:47.729811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.686319    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.788689    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.788838    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.788910    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.788957    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.789034    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvn9d\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.789094    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.789131    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.789171    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.801918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.789256    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.812201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.811724    4295 generic.go:356] "Generic (PLEG): container finished" podID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerID="99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e" exitCode=0
Oct 30 22:15:47.812201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.811844    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerDied","Data":"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"}
Oct 30 22:15:47.813022 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.812968    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-516ccb-openshift-default-dockercfg-w96vw\""
Oct 30 22:15:47.864817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.815770    4295 scope.go:122] "RemoveContainer" containerID="99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"
Oct 30 22:15:47.864817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.848513    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:47.864817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.848637    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:47.864817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.848720    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:47.917029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.910772    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.947688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.917411    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.948516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.948482    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvn9d\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.949258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.949230    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.949908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.949891    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.955683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.950312    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.912958    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.950255    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.958433    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.958527    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.949824    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.954828    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.952321    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.958894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.958738    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.959472 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.959452    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:47.961678 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:47.961627    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:48.012485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.012458    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvn9d\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:48.027408 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.027376    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\") pod \"e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") " pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:48.112016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.111987    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:48.757396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.757360    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:15:48.816702 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:48.816115    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8070ff_d9cf_44c4_a3c1_466f1e64e751.slice/crio-efe53e01f661c86334d608f2e2bd43fede2fba33958390a8dff0e83d493fcabb WatchSource:0}: Error finding container efe53e01f661c86334d608f2e2bd43fede2fba33958390a8dff0e83d493fcabb: Status 404 returned error can't find the container with id efe53e01f661c86334d608f2e2bd43fede2fba33958390a8dff0e83d493fcabb
Oct 30 22:15:48.819196 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.819156    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:48.819414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.819386    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:48.819510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.819465    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:48.826795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:48.826761    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerStarted","Data":"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"}
Oct 30 22:15:49.514643 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.514604    4295 patch_prober.go:29] interesting pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:15:49.514856 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.514691    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:15:49.518182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.518160    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:15:49.834172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.834077    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerStarted","Data":"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"}
Oct 30 22:15:49.834172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.834148    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerStarted","Data":"efe53e01f661c86334d608f2e2bd43fede2fba33958390a8dff0e83d493fcabb"}
Oct 30 22:15:49.960080 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:49.960027    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podStartSLOduration=2.96001274 podStartE2EDuration="2.96001274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:49.957051303 +0000 UTC m=+1552.599645092" watchObservedRunningTime="2026-10-30 22:15:49.96001274 +0000 UTC m=+1552.602606530"
Oct 30 22:15:50.119166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.119082    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:50.161019 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.160986    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:50.505279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.505242    4295 patch_prober.go:29] interesting pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:15:50.505576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.505549    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:15:50.841067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.840982    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:50.842108 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:50.842089    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:15:51.512731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:51.511774    4295 patch_prober.go:29] interesting pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:15:51.512731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:51.511837    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:15:52.513121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:52.513084    4295 patch_prober.go:29] interesting pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:15:52.513784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:52.513748    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:15:52.880079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:52.880011    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:15:53.506766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:53.506727    4295 patch_prober.go:29] interesting pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:15:53.507049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:53.507020    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:15:53.634353 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:15:53.634316    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:15:54.507859 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:54.507832    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:15:54.508748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:54.508732    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:15:58.516692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.515794    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:15:58.585684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.581977    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:58.585684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.585253    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:15:58.718318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.716904    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-516ccb-dockercfg-tr9pd\""
Oct 30 22:15:58.718712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.718682    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-516ccb\""
Oct 30 22:15:58.790389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.788275    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:15:58.803687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.801894    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:15:58.803687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.802015    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:58.815724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.815298    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:58.816018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.815992    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4bb\" (UniqueName: \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:58.965710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.961350    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msz7\" (UniqueName: \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:58.965710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.961473    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:58.965710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.961514    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4bb\" (UniqueName: \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:58.965710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.961641    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:58.966042 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:58.966005    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:59.049176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.048758    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4bb\" (UniqueName: \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\") pod \"payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:59.062818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.062791    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:59.062951 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.062930    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msz7\" (UniqueName: \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:59.063458 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.063435    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:59.128166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.128139    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msz7\" (UniqueName: \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\") pod \"payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:59.196575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.196539    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:15:59.215015 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.214969    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:15:59.582242 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.582153    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:15:59.618167 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:59.618143    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbf3240_f276_4e1d_9b02_9a43a95f09bc.slice/crio-28a779cc1d40875bedf004bce04784df93f01f592cb2e386a4381358b08788e3 WatchSource:0}: Error finding container 28a779cc1d40875bedf004bce04784df93f01f592cb2e386a4381358b08788e3: Status 404 returned error can't find the container with id 28a779cc1d40875bedf004bce04784df93f01f592cb2e386a4381358b08788e3
Oct 30 22:15:59.634134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:15:59.634111    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:15:59.635792 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:15:59.635769    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa188cd6_41c9_4b34_b7c0_8de7673c7136.slice/crio-1e0c720f7c490dae89c2ce23a729bc40ec336a759f37402cd59112909e6c270e WatchSource:0}: Error finding container 1e0c720f7c490dae89c2ce23a729bc40ec336a759f37402cd59112909e6c270e: Status 404 returned error can't find the container with id 1e0c720f7c490dae89c2ce23a729bc40ec336a759f37402cd59112909e6c270e
Oct 30 22:16:00.023273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.023239    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" event={"ID":"0fbf3240-f276-4e1d-9b02-9a43a95f09bc","Type":"ContainerStarted","Data":"8a23b996a5ae4bc287478771045ce9b496c0154dc8dd281f638ae9173b6ec5f8"}
Oct 30 22:16:00.023475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.023330    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:16:00.023475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.023349    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" event={"ID":"0fbf3240-f276-4e1d-9b02-9a43a95f09bc","Type":"ContainerStarted","Data":"28a779cc1d40875bedf004bce04784df93f01f592cb2e386a4381358b08788e3"}
Oct 30 22:16:00.024922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.024892    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" event={"ID":"aa188cd6-41c9-4b34-b7c0-8de7673c7136","Type":"ContainerStarted","Data":"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"}
Oct 30 22:16:00.025049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.024930    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" event={"ID":"aa188cd6-41c9-4b34-b7c0-8de7673c7136","Type":"ContainerStarted","Data":"1e0c720f7c490dae89c2ce23a729bc40ec336a759f37402cd59112909e6c270e"}
Oct 30 22:16:00.025049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.025014    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:16:00.144123 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.144069    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" podStartSLOduration=2.144056181 podStartE2EDuration="2.144056181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:00.143089324 +0000 UTC m=+1562.785683115" watchObservedRunningTime="2026-10-30 22:16:00.144056181 +0000 UTC m=+1562.786649978"
Oct 30 22:16:00.293744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:00.292477    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" podStartSLOduration=2.292457686 podStartE2EDuration="2.292457686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:00.290631698 +0000 UTC m=+1562.933225487" watchObservedRunningTime="2026-10-30 22:16:00.292457686 +0000 UTC m=+1562.935051477"
Oct 30 22:16:06.191603 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.191571    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:16:06.195956 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.195935    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.255851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.255820    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:16:06.302117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.302085    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7422b4b3-serving-cert\""
Oct 30 22:16:06.309964 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.309933    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.310060 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.309978    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78hf\" (UniqueName: \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.310187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.310167    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.310291 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.310275    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.411821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.411781    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.411990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.411916    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.411990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.411956    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78hf\" (UniqueName: \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.412138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.412123    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.412284 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:06.412264    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-serving-cert: secret "maas-api-e2e-mt-7422b4b3-serving-cert" not found
Oct 30 22:16:06.412361 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:06.412341    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls podName:7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254 nodeName:}" failed. No retries permitted until 2026-10-30 22:16:06.912320993 +0000 UTC m=+1569.554914762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls") pod "maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" (UID: "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254") : secret "maas-api-e2e-mt-7422b4b3-serving-cert" not found
Oct 30 22:16:06.412551 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.412530    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.414852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.414832    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.513733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.513049    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78hf\" (UniqueName: \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.927331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.927245    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:06.929902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:06.929880    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") pod \"maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:07.110779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:07.110740    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:07.505575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:07.500459    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:16:07.506511 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:16:07.506479    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2cd0af_2ab1_4b60_8cf7_47e3e4e21254.slice/crio-08feb071bb00d3bd3121f0f452ee2a0fb90b6b8d471bd4443a8c8f1bfd3bf00a WatchSource:0}: Error finding container 08feb071bb00d3bd3121f0f452ee2a0fb90b6b8d471bd4443a8c8f1bfd3bf00a: Status 404 returned error can't find the container with id 08feb071bb00d3bd3121f0f452ee2a0fb90b6b8d471bd4443a8c8f1bfd3bf00a
Oct 30 22:16:08.076917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:08.076870    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" event={"ID":"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254","Type":"ContainerStarted","Data":"08feb071bb00d3bd3121f0f452ee2a0fb90b6b8d471bd4443a8c8f1bfd3bf00a"}
Oct 30 22:16:09.085050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:09.084964    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" event={"ID":"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254","Type":"ContainerStarted","Data":"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"}
Oct 30 22:16:09.085408 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:09.085196    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:09.211269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:09.211216    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" podStartSLOduration=2.382438255 podStartE2EDuration="3.211201458s" totalImagesPullingTime="828.763203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:09.206981455 +0000 UTC m=+1571.849575243" watchObservedRunningTime="2026-10-30 22:16:09.211201458 +0000 UTC m=+1571.853795248"
Oct 30 22:16:11.034339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:11.034310    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:16:11.034748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:11.034375    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:16:15.098839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:15.098811    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:16:18.516694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.516504    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:16:18.523642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.523613    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.647591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.647533    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ef8140e-openshift-default-dockercfg-24rvz\""
Oct 30 22:16:18.647997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.647962    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.648215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.648089    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.648215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.648168    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.648548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.648375    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.648648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.648560    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.648648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.648617    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.649736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.649708    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.649855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.649750    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7p9\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.649855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.649796    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.653932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.653912    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:16:18.750723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750680    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.750921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750775    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.750921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750819    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.750921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750857    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.750921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750905    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750933    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750963    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.750989    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7p9\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751041    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751337    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751430    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751460    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751709    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.751779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.751728    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.753301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.753276    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.753720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.753700    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.817943 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.817862    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7p9\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.822974 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.822949    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:18.839005 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:18.838972    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:19.275301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.275270    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:16:19.275421 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:16:19.275348    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf971d96b_f399_4825_8dfa_83a581ece4dc.slice/crio-7d49553076ecd5e7f6ee95b4ec1f44a6a7a39fc28e9edfb6bc0f94515e930608 WatchSource:0}: Error finding container 7d49553076ecd5e7f6ee95b4ec1f44a6a7a39fc28e9edfb6bc0f94515e930608: Status 404 returned error can't find the container with id 7d49553076ecd5e7f6ee95b4ec1f44a6a7a39fc28e9edfb6bc0f94515e930608
Oct 30 22:16:19.278107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.278075    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:19.278161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.278138    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:19.278203 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.278170    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:19.939690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.939508    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:16:19.965688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:19.962298    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.032695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.028400    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-422c1029-openshift-default-dockercfg-f8xcc\""
Oct 30 22:16:20.122690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.100436    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.125958    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149059    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149207    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149287    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149324    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149366    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149405    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149558    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.150114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.149631    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.218111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.218015    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" event={"ID":"f971d96b-f399-4825-8dfa-83a581ece4dc","Type":"ContainerStarted","Data":"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"}
Oct 30 22:16:20.218388 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.218359    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" event={"ID":"f971d96b-f399-4825-8dfa-83a581ece4dc","Type":"ContainerStarted","Data":"7d49553076ecd5e7f6ee95b4ec1f44a6a7a39fc28e9edfb6bc0f94515e930608"}
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.254796    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.254883    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.254943    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255040    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255161    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255241    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255277    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255316    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255350    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.255967    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.256391    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.259799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.256985    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.317972 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.317920    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.318558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.318521    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.318712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.318616    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.352357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.352326    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.383201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.383166    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.401635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.399411    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.469079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.468988    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:20.839531 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.839447    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:20.841136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.841110    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:20.841272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:20.841166    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:21.018567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.018524    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podStartSLOduration=3.018507858 podStartE2EDuration="3.018507858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:20.399795188 +0000 UTC m=+1583.042388976" watchObservedRunningTime="2026-10-30 22:16:21.018507858 +0000 UTC m=+1583.661101650"
Oct 30 22:16:21.018917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.018704    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:16:21.020605 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:16:21.020562    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd26d361_a47d_4531_a0e5_74ee9b9b7a05.slice/crio-9340e6bdea2704ba098fb7dff5f3c0c7713d30f1c63ee131568bb33894e02687 WatchSource:0}: Error finding container 9340e6bdea2704ba098fb7dff5f3c0c7713d30f1c63ee131568bb33894e02687: Status 404 returned error can't find the container with id 9340e6bdea2704ba098fb7dff5f3c0c7713d30f1c63ee131568bb33894e02687
Oct 30 22:16:21.023419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.023385    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.023499 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.023452    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.023499 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.023480    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.226348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.226314    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerStarted","Data":"55fe89166a1afe1ca22857b31673892c6a64231e72d7199dbd77444efa983f60"}
Oct 30 22:16:21.226555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.226371    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerStarted","Data":"9340e6bdea2704ba098fb7dff5f3c0c7713d30f1c63ee131568bb33894e02687"}
Oct 30 22:16:21.353052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.353002    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podStartSLOduration=2.352988696 podStartE2EDuration="2.352988696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:21.351587184 +0000 UTC m=+1583.994180973" watchObservedRunningTime="2026-10-30 22:16:21.352988696 +0000 UTC m=+1583.995582485"
Oct 30 22:16:21.470161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.470128    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:21.843268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.840263    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:21.843268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:21.840341    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:22.475711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:22.475645    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:22.839573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:22.839490    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:22.839573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:22.839544    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:23.317014 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:23.315932    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:23.330030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:23.329985    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:16:23.852715 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:23.847081    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:23.859241 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:23.858793    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:23.883080 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:23.883041    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:16:24.839620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:24.839583    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:24.839865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:24.839647    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:25.840013 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:25.839976    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:25.840438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:25.840038    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:16:26.843851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:26.843820    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:26.844265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:26.844170    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:26.844932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:26.844914    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:16:28.516693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.499183    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:16:28.516693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.513511    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:28.571141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.571099    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:16:28.599157 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.599026    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-422c1029-dockercfg-8gd98\""
Oct 30 22:16:28.599157 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.599126    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-422c1029\""
Oct 30 22:16:28.718480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.718430    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:28.718706 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.718544    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cmd\" (UniqueName: \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:28.976976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.976363    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cmd\" (UniqueName: \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:28.976976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:28.976790    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:29.037523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:29.037483    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:29.118480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:29.118451    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cmd\" (UniqueName: \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:29.179814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:29.179783    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:29.617469 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:16:29.617435    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4613974b_3aad_4e2c_b8cb_fa3dd2b5efdd.slice/crio-8076d34a5fa0e3645d041fe24d687a8d8897900a5de536209d0b6dcd45dd51bd WatchSource:0}: Error finding container 8076d34a5fa0e3645d041fe24d687a8d8897900a5de536209d0b6dcd45dd51bd: Status 404 returned error can't find the container with id 8076d34a5fa0e3645d041fe24d687a8d8897900a5de536209d0b6dcd45dd51bd
Oct 30 22:16:29.619171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:29.619143    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:16:30.530076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:30.530034    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" event={"ID":"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd","Type":"ContainerStarted","Data":"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"}
Oct 30 22:16:30.530369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:30.530345    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" event={"ID":"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd","Type":"ContainerStarted","Data":"8076d34a5fa0e3645d041fe24d687a8d8897900a5de536209d0b6dcd45dd51bd"}
Oct 30 22:16:30.532543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:30.532521    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:30.721727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:30.710478    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" podStartSLOduration=2.710457516 podStartE2EDuration="2.710457516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:30.681780464 +0000 UTC m=+1593.324374254" watchObservedRunningTime="2026-10-30 22:16:30.710457516 +0000 UTC m=+1593.353051306"
Oct 30 22:16:32.536873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:32.536837    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:16:37.537247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.537179    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" podUID="90c9bd0f-03c6-48fd-9916-cedc44583245" containerName="payload-pre-processing" containerID="cri-o://6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db" gracePeriod=25
Oct 30 22:16:37.820191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.820162    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:16:37.879779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.879740    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\") pod \"90c9bd0f-03c6-48fd-9916-cedc44583245\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") "
Oct 30 22:16:37.879984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.879829    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\" (UniqueName: \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\") pod \"90c9bd0f-03c6-48fd-9916-cedc44583245\" (UID: \"90c9bd0f-03c6-48fd-9916-cedc44583245\") "
Oct 30 22:16:37.880183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.880156    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume" pod "90c9bd0f-03c6-48fd-9916-cedc44583245" (UID: "90c9bd0f-03c6-48fd-9916-cedc44583245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:37.880647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.880626    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c9bd0f-03c6-48fd-9916-cedc44583245-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:37.883114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.883082    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d" pod "90c9bd0f-03c6-48fd-9916-cedc44583245" (UID: "90c9bd0f-03c6-48fd-9916-cedc44583245"). InnerVolumeSpecName "kube-api-access-zxm5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:37.983015 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:37.982976    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxm5d\" (UniqueName: \"kubernetes.io/projected/90c9bd0f-03c6-48fd-9916-cedc44583245-kube-api-access-zxm5d\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.598903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.598786    4295 generic.go:356] "Generic (PLEG): container finished" podID="90c9bd0f-03c6-48fd-9916-cedc44583245" containerID="6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db" exitCode=0
Oct 30 22:16:38.598903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.598822    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" event={"ID":"90c9bd0f-03c6-48fd-9916-cedc44583245","Type":"ContainerDied","Data":"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"}
Oct 30 22:16:38.598903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.598844    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"
Oct 30 22:16:38.598903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.598877    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms" event={"ID":"90c9bd0f-03c6-48fd-9916-cedc44583245","Type":"ContainerDied","Data":"a8b43bcbe985f953d23b30a8cf8be185bf5126f15814f88b6f505c90e9c18913"}
Oct 30 22:16:38.598903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.598889    4295 scope.go:122] "RemoveContainer" containerID="6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"
Oct 30 22:16:38.600657 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.600635    4295 generic.go:356] "Generic (PLEG): container finished" podID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerID="3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9" exitCode=0
Oct 30 22:16:38.600787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.600719    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerDied","Data":"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"}
Oct 30 22:16:38.601218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.601202    4295 scope.go:122] "RemoveContainer" containerID="3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"
Oct 30 22:16:38.602967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.602935    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.603081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.603014    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.603081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.603054    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.617008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.616989    4295 scope.go:122] "RemoveContainer" containerID="6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"
Oct 30 22:16:38.617329 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:38.617308    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db\": container with ID starting with 6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db not found: ID does not exist" containerID="6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"
Oct 30 22:16:38.617437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.617340    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db"} err="failed to get container status \"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db\": rpc error: code = NotFound desc = could not find container \"6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db\": container with ID starting with 6d302ff17b441e2fd6a25d34f79f22b7becc0f3f53f388c400233054c2c4f1db not found: ID does not exist"
Oct 30 22:16:38.792083 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.792030    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:16:38.796552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:38.796428    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms"]
Oct 30 22:16:39.517717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:39.517640    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c9bd0f-03c6-48fd-9916-cedc44583245" path="/var/lib/kubelet/pods/90c9bd0f-03c6-48fd-9916-cedc44583245/volumes"
Oct 30 22:16:39.620514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:39.620480    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerStarted","Data":"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"}
Oct 30 22:16:40.248930 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:40.248897    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:16:40.249981 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:40.249957    4295 patch_prober.go:29] interesting pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:16:40.250071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:40.250013    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:16:41.262694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:41.252117    4295 patch_prober.go:29] interesting pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:16:41.262694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:41.252188    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:16:42.249356 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:42.249317    4295 patch_prober.go:29] interesting pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:16:42.249540 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:42.249369    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:16:42.545468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:42.545376    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:16:43.249772 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:43.249732    4295 patch_prober.go:29] interesting pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:16:43.249961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:43.249813    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:16:43.459588 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:43.459553    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:16:44.248615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:44.248575    4295 patch_prober.go:29] interesting pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:16:44.249029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:44.248648    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:16:45.253909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:45.253881    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:16:45.255114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:45.255097    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:16:52.395377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:52.395343    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:16:52.395885 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:52.395586    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy" containerID="cri-o://aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194" gracePeriod=30
Oct 30 22:16:53.609257 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:53.609213    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:16:57.682265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.682241    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:16:57.727638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727605    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.727813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727746    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.727813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727784    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.727934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727864    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.727934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727886    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.727934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727897    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:57.728086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727941    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.728086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.727985    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.728174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728073    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:57.728174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728122    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.728292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728156    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:57.728292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728178    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\") pod \"006bff34-f902-46d0-8469-543ae5eced45\" (UID: \"006bff34-f902-46d0-8469-543ae5eced45\") "
Oct 30 22:16:57.728400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728375    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:57.728441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.728408    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:57.729189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.729163    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.729316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.729192    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.729316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.729207    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006bff34-f902-46d0-8469-543ae5eced45-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.729316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.729221    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.729316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.729236    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.730424 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.730402    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:57.730508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.730469    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:57.730547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.730529    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:57.730585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.730561    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889" pod "006bff34-f902-46d0-8469-543ae5eced45" (UID: "006bff34-f902-46d0-8469-543ae5eced45"). InnerVolumeSpecName "kube-api-access-nt889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:57.744419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.744395    4295 generic.go:356] "Generic (PLEG): container finished" podID="006bff34-f902-46d0-8469-543ae5eced45" containerID="aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194" exitCode=0
Oct 30 22:16:57.744517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.744493    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"
Oct 30 22:16:57.744557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.744497    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" event={"ID":"006bff34-f902-46d0-8469-543ae5eced45","Type":"ContainerDied","Data":"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"}
Oct 30 22:16:57.744602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.744571    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l" event={"ID":"006bff34-f902-46d0-8469-543ae5eced45","Type":"ContainerDied","Data":"be6385b9e261be96e3c2772541d76650fd7bbd14f60e86b92c57df8863d0425b"}
Oct 30 22:16:57.744710 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.744624    4295 scope.go:122] "RemoveContainer" containerID="aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"
Oct 30 22:16:57.759960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.759938    4295 scope.go:122] "RemoveContainer" containerID="aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"
Oct 30 22:16:57.760279 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:16:57.760260    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194\": container with ID starting with aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194 not found: ID does not exist" containerID="aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"
Oct 30 22:16:57.760342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.760289    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194"} err="failed to get container status \"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194\": rpc error: code = NotFound desc = could not find container \"aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194\": container with ID starting with aa11b357f1c1caa8953cf0615955ba3451d38814fe1f86dd0aef0ae4edfb1194 not found: ID does not exist"
Oct 30 22:16:57.830496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.830462    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006bff34-f902-46d0-8469-543ae5eced45-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.830496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.830491    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006bff34-f902-46d0-8469-543ae5eced45-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.830496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.830502    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt889\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-kube-api-access-nt889\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.830744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.830511    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006bff34-f902-46d0-8469-543ae5eced45-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.955917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.955883    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:16:57.963823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:57.963793    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l"]
Oct 30 22:16:59.495268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:16:59.495236    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006bff34-f902-46d0-8469-543ae5eced45" path="/var/lib/kubelet/pods/006bff34-f902-46d0-8469-543ae5eced45/volumes"
Oct 30 22:17:03.863817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.863781    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:03.864407 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864393    4295 cpu_manager.go:414] "containerMap: removing container" podUID="90c9bd0f-03c6-48fd-9916-cedc44583245" containerName="payload-pre-processing"
Oct 30 22:17:03.864454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864408    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c9bd0f-03c6-48fd-9916-cedc44583245" containerName="payload-pre-processing"
Oct 30 22:17:03.864454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864447    4295 cpu_manager.go:414] "containerMap: removing container" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy"
Oct 30 22:17:03.864454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864452    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy"
Oct 30 22:17:03.864557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864551    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="006bff34-f902-46d0-8469-543ae5eced45" containerName="istio-proxy"
Oct 30 22:17:03.864590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.864561    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c9bd0f-03c6-48fd-9916-cedc44583245" containerName="payload-pre-processing"
Oct 30 22:17:03.869461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.869447    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929534    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929580    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929615    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929681    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5x7\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929761    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929940 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929810    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.929987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.929962    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.930039 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.930024    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:03.930140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:03.930127    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.007031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.006995    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:04.031535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031504    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.031755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031554    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.031755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031625    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.031755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031712    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.031755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031740    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031775    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031800    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5x7\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031845    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.031887    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.032030    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.032066    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.032335    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.032349    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.032881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.032853    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.035267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.035242    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.035487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.035469    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.096344 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.096314    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.097118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.097095    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5x7\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.182974 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.182945    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:04.618433 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:17:04.618405    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1733c0cf_3922_48fb_aa68_0c9ffbeb329e.slice/crio-1055bf4714c1693f45d4d6af3cb0cee49f8b1d8b2b2c44e0234f25e78985cbe6 WatchSource:0}: Error finding container 1055bf4714c1693f45d4d6af3cb0cee49f8b1d8b2b2c44e0234f25e78985cbe6: Status 404 returned error can't find the container with id 1055bf4714c1693f45d4d6af3cb0cee49f8b1d8b2b2c44e0234f25e78985cbe6
Oct 30 22:17:04.618552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.618454    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:04.620975 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.620945    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.621030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.621008    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.621067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.621041    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.796171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.796129    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" event={"ID":"1733c0cf-3922-48fb-aa68-0c9ffbeb329e","Type":"ContainerStarted","Data":"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"}
Oct 30 22:17:04.796171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.796179    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" event={"ID":"1733c0cf-3922-48fb-aa68-0c9ffbeb329e","Type":"ContainerStarted","Data":"1055bf4714c1693f45d4d6af3cb0cee49f8b1d8b2b2c44e0234f25e78985cbe6"}
Oct 30 22:17:04.921832 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:04.921719    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podStartSLOduration=1.921704949 podStartE2EDuration="1.921704949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:04.920233639 +0000 UTC m=+1627.562827431" watchObservedRunningTime="2026-10-30 22:17:04.921704949 +0000 UTC m=+1627.564298737"
Oct 30 22:17:05.185904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:05.185849    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:05.188186 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:05.188154    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.188434 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:05.188409    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:06.183856 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:06.183818    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.184249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:06.183875    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:06.462881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:06.462799    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:17:06.463058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:06.463033    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" podUID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" containerName="maas-api" containerID="cri-o://c9b6da653c4db5f7450541ce731b448ebc56bc3a1ad518820da85f83c446fbe0" gracePeriod=30
Oct 30 22:17:07.207369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.207330    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.207999 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.207958    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:07.214745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.214714    4295 generic.go:356] "Generic (PLEG): container finished" podID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" containerID="c9b6da653c4db5f7450541ce731b448ebc56bc3a1ad518820da85f83c446fbe0" exitCode=0
Oct 30 22:17:07.216042 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.215987    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" event={"ID":"15c68ef0-ef33-4195-b1c2-93c3e67107cf","Type":"ContainerDied","Data":"c9b6da653c4db5f7450541ce731b448ebc56bc3a1ad518820da85f83c446fbe0"}
Oct 30 22:17:07.424112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.424080    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:17:07.533738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.523027    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") pod \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") "
Oct 30 22:17:07.586110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.584384    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls" pod "15c68ef0-ef33-4195-b1c2-93c3e67107cf" (UID: "15c68ef0-ef33-4195-b1c2-93c3e67107cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:07.648701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.646957    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\") pod \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") "
Oct 30 22:17:07.648701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.647895    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca" pod "15c68ef0-ef33-4195-b1c2-93c3e67107cf" (UID: "15c68ef0-ef33-4195-b1c2-93c3e67107cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:07.676689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.647031    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\") pod \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") "
Oct 30 22:17:07.676689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.670745    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\" (UniqueName: \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\") pod \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\" (UID: \"15c68ef0-ef33-4195-b1c2-93c3e67107cf\") "
Oct 30 22:17:07.755918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.717614    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:07.755918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.717970    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c68ef0-ef33-4195-b1c2-93c3e67107cf-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:07.755918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.755068    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n" pod "15c68ef0-ef33-4195-b1c2-93c3e67107cf" (UID: "15c68ef0-ef33-4195-b1c2-93c3e67107cf"). InnerVolumeSpecName "kube-api-access-qhd7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:07.760467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.760426    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs" pod "15c68ef0-ef33-4195-b1c2-93c3e67107cf" (UID: "15c68ef0-ef33-4195-b1c2-93c3e67107cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:07.829745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.826106    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhd7n\" (UniqueName: \"kubernetes.io/projected/15c68ef0-ef33-4195-b1c2-93c3e67107cf-kube-api-access-qhd7n\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:07.829745 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:07.826149    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c68ef0-ef33-4195-b1c2-93c3e67107cf-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:08.183959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.183927    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:08.184121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.184000    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:08.223758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.223726    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4" event={"ID":"15c68ef0-ef33-4195-b1c2-93c3e67107cf","Type":"ContainerDied","Data":"3a91d0fd64e65e750e1f6b968695881a963936330c198bba8f8b4482edd82aec"}
Oct 30 22:17:08.223758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.223758    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"
Oct 30 22:17:08.224324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.223797    4295 scope.go:122] "RemoveContainer" containerID="c9b6da653c4db5f7450541ce731b448ebc56bc3a1ad518820da85f83c446fbe0"
Oct 30 22:17:08.416315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.416273    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:17:08.420064 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:08.420040    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ef8140e-f664cc44f-f46z4"]
Oct 30 22:17:09.183613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:09.183563    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:09.183846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:09.183616    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:09.523910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:09.523824    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" path="/var/lib/kubelet/pods/15c68ef0-ef33-4195-b1c2-93c3e67107cf/volumes"
Oct 30 22:17:10.183375 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.183336    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:17:10.183538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.183401    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 22:17:10.422113 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.422058    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:10.422494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.422445    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy" containerID="cri-o://8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e" gracePeriod=30
Oct 30 22:17:10.426096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.426070    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:17:10.426263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:10.426243    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" containerID="cri-o://7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2" gracePeriod=30
Oct 30 22:17:11.846867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:11.846819    4295 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:11.847337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:11.846926    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:13.514650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:13.514621    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:17:13.515041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:13.514816    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" podUID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" containerName="maas-api" containerID="cri-o://2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72" gracePeriod=30
Oct 30 22:17:14.388643 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.388566    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:17:14.393506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.393473    4295 generic.go:356] "Generic (PLEG): container finished" podID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" containerID="2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72" exitCode=0
Oct 30 22:17:14.393694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.393530    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"
Oct 30 22:17:14.393694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.393551    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" event={"ID":"df00b75d-4ed8-4ad2-abc4-da8f0a309f83","Type":"ContainerDied","Data":"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"}
Oct 30 22:17:14.393694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.393612    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59" event={"ID":"df00b75d-4ed8-4ad2-abc4-da8f0a309f83","Type":"ContainerDied","Data":"7ef9a909e46d38179406213f1efb851c2f963db2aa2203c94e7cb0449ccfd604"}
Oct 30 22:17:14.393694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.393646    4295 scope.go:122] "RemoveContainer" containerID="2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"
Oct 30 22:17:14.404432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.404412    4295 scope.go:122] "RemoveContainer" containerID="2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"
Oct 30 22:17:14.404752 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:14.404728    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72\": container with ID starting with 2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72 not found: ID does not exist" containerID="2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"
Oct 30 22:17:14.404830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.404759    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72"} err="failed to get container status \"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72\": rpc error: code = NotFound desc = could not find container \"2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72\": container with ID starting with 2e447eaed93b77e47ad282bf5b173c4e7e3371f3cc1406f88a7c307c30130b72 not found: ID does not exist"
Oct 30 22:17:14.480912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.480875    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\" (UniqueName: \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\") pod \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") "
Oct 30 22:17:14.481070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.480975    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\") pod \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") "
Oct 30 22:17:14.481070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.481050    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") pod \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") "
Oct 30 22:17:14.481201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.481103    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\") pod \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\" (UID: \"df00b75d-4ed8-4ad2-abc4-da8f0a309f83\") "
Oct 30 22:17:14.481591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.481562    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca" pod "df00b75d-4ed8-4ad2-abc4-da8f0a309f83" (UID: "df00b75d-4ed8-4ad2-abc4-da8f0a309f83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:14.481771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.481732    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.483260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.483240    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls" pod "df00b75d-4ed8-4ad2-abc4-da8f0a309f83" (UID: "df00b75d-4ed8-4ad2-abc4-da8f0a309f83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:14.483646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.483631    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs" pod "df00b75d-4ed8-4ad2-abc4-da8f0a309f83" (UID: "df00b75d-4ed8-4ad2-abc4-da8f0a309f83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:14.483718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.483700    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d" pod "df00b75d-4ed8-4ad2-abc4-da8f0a309f83" (UID: "df00b75d-4ed8-4ad2-abc4-da8f0a309f83"). InnerVolumeSpecName "kube-api-access-6bn6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:14.584311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.584274    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.584311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.584312    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.584766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.584329    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn6d\" (UniqueName: \"kubernetes.io/projected/df00b75d-4ed8-4ad2-abc4-da8f0a309f83-kube-api-access-6bn6d\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.952257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.952157    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:17:14.952516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:14.952494    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-781ccf27-5664496bfc-95r59"]
Oct 30 22:17:15.496201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.496162    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" path="/var/lib/kubelet/pods/df00b75d-4ed8-4ad2-abc4-da8f0a309f83/volumes"
Oct 30 22:17:15.705553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.705499    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865088    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865175    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865223    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865288    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865457    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865500    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.874687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.865591    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.880684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.877975    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.880684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.878142    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.881262    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:15.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.881510    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.881706    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\") pod \"f971d96b-f399-4825-8dfa-83a581ece4dc\" (UID: \"f971d96b-f399-4825-8dfa-83a581ece4dc\") "
Oct 30 22:17:15.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.881876    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.896684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.882123    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.899312 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.899231    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971d96b-f399-4825-8dfa-83a581ece4dc-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.899312 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.899262    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.899312 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.899276    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.899517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.899290    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.899517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.899401    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.920723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.917819    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:15.920723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.918004    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:15.920723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.918565    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.971912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:15.964098    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9" pod "f971d96b-f399-4825-8dfa-83a581ece4dc" (UID: "f971d96b-f399-4825-8dfa-83a581ece4dc"). InnerVolumeSpecName "kube-api-access-rp7p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:16.001441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.001403    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:16.001441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.001442    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7p9\" (UniqueName: \"kubernetes.io/projected/f971d96b-f399-4825-8dfa-83a581ece4dc-kube-api-access-rp7p9\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:16.001441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.001457    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:16.001839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.001471    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971d96b-f399-4825-8dfa-83a581ece4dc-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:16.178544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.178514    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:17:16.178793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.178757    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy" containerID="cri-o://eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863" gracePeriod=30
Oct 30 22:17:16.478446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.478409    4295 generic.go:356] "Generic (PLEG): container finished" podID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerID="7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2" exitCode=0
Oct 30 22:17:16.479048 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.478587    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"
Oct 30 22:17:16.483482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.483291    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" event={"ID":"f971d96b-f399-4825-8dfa-83a581ece4dc","Type":"ContainerDied","Data":"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"}
Oct 30 22:17:16.483482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.483375    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt" event={"ID":"f971d96b-f399-4825-8dfa-83a581ece4dc","Type":"ContainerDied","Data":"7d49553076ecd5e7f6ee95b4ec1f44a6a7a39fc28e9edfb6bc0f94515e930608"}
Oct 30 22:17:16.483482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.483407    4295 scope.go:122] "RemoveContainer" containerID="7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"
Oct 30 22:17:16.562123 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.562087    4295 scope.go:122] "RemoveContainer" containerID="7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"
Oct 30 22:17:16.562515 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:16.562473    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2\": container with ID starting with 7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2 not found: ID does not exist" containerID="7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"
Oct 30 22:17:16.562585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.562529    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2"} err="failed to get container status \"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2\": rpc error: code = NotFound desc = could not find container \"7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2\": container with ID starting with 7039c87f7aab0603454e432b34089b89d7ab0c523c18a7b9e215fe5f185e33b2 not found: ID does not exist"
Oct 30 22:17:16.660927 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.660894    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:17:16.669278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.669247    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt"]
Oct 30 22:17:16.892605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:16.892581    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:17.062226 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062129    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062226 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062170    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062226 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062195    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062237    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062428    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.062520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062477    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062521    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062557    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062616    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062720    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\") pod \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\" (UID: \"1733c0cf-3922-48fb-aa68-0c9ffbeb329e\") "
Oct 30 22:17:17.062729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062722    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:17.062992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.062723    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.063055 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063011    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.063435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063364    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.063742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063459    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.063742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063590    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.063742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063604    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.063742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.063618    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.065181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.065078    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.065181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.065144    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.065181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.065150    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "kube-api-access-hj5x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.065403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.065385    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo" pod "1733c0cf-3922-48fb-aa68-0c9ffbeb329e" (UID: "1733c0cf-3922-48fb-aa68-0c9ffbeb329e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:17.165432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.165402    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.165432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.165426    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.165432 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.165436    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.165633 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.165445    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.165633 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.165454    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5x7\" (UniqueName: \"kubernetes.io/projected/1733c0cf-3922-48fb-aa68-0c9ffbeb329e-kube-api-access-hj5x7\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.494886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.494326    4295 generic.go:356] "Generic (PLEG): container finished" podID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerID="8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e" exitCode=0
Oct 30 22:17:17.494886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.494490    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"
Oct 30 22:17:17.500582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.500551    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" path="/var/lib/kubelet/pods/f971d96b-f399-4825-8dfa-83a581ece4dc/volumes"
Oct 30 22:17:17.501157 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.501132    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" event={"ID":"1733c0cf-3922-48fb-aa68-0c9ffbeb329e","Type":"ContainerDied","Data":"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"}
Oct 30 22:17:17.501254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.501186    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g" event={"ID":"1733c0cf-3922-48fb-aa68-0c9ffbeb329e","Type":"ContainerDied","Data":"1055bf4714c1693f45d4d6af3cb0cee49f8b1d8b2b2c44e0234f25e78985cbe6"}
Oct 30 22:17:17.501254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.501219    4295 scope.go:122] "RemoveContainer" containerID="8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"
Oct 30 22:17:17.513530 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.513510    4295 scope.go:122] "RemoveContainer" containerID="8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"
Oct 30 22:17:17.513865 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:17.513842    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e\": container with ID starting with 8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e not found: ID does not exist" containerID="8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"
Oct 30 22:17:17.513941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.513875    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e"} err="failed to get container status \"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e\": rpc error: code = NotFound desc = could not find container \"8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e\": container with ID starting with 8c54635be6f8474c7f6728ff205c050a5de334f21987e1830d02f3fcbffed49e not found: ID does not exist"
Oct 30 22:17:17.763737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.756428    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:17.794691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:17.773553    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g"]
Oct 30 22:17:19.557580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:19.557545    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" path="/var/lib/kubelet/pods/1733c0cf-3922-48fb-aa68-0c9ffbeb329e/volumes"
Oct 30 22:17:21.476679 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.476636    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:17:21.606523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606492    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606558    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606592    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606624    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606649    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606749    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606826    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.606953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606900    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.607121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.606952    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\") pod \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\" (UID: \"544e6d2e-ba0f-4029-b7c8-a45fe75e2770\") "
Oct 30 22:17:21.607121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607065    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:21.607121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607083    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:21.607272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607235    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:21.607487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607367    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:21.607766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607743    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.607849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607771    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.607849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607787    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.607849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607800    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.607997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.607926    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:21.608991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.608965    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:21.609103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.609083    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:21.609418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.609395    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:21.609522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.609426    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks" pod "544e6d2e-ba0f-4029-b7c8-a45fe75e2770" (UID: "544e6d2e-ba0f-4029-b7c8-a45fe75e2770"). InnerVolumeSpecName "kube-api-access-b2zks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:21.709705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.709610    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2zks\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-kube-api-access-b2zks\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.709705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.709643    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.709705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.709654    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.709705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.709700    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.709953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.709714    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544e6d2e-ba0f-4029-b7c8-a45fe75e2770-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.794004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.793970    4295 generic.go:356] "Generic (PLEG): container finished" podID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerID="eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863" exitCode=0
Oct 30 22:17:21.794179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.794021    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"
Oct 30 22:17:21.794179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.794052    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerDied","Data":"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"}
Oct 30 22:17:21.794179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.794110    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d" event={"ID":"544e6d2e-ba0f-4029-b7c8-a45fe75e2770","Type":"ContainerDied","Data":"a22a185242543e7d42be8459fbf3a3a0d5ada4c8e5a1ea75d8157c0da8a78fb2"}
Oct 30 22:17:21.794179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.794136    4295 scope.go:122] "RemoveContainer" containerID="eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"
Oct 30 22:17:21.804837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.804820    4295 scope.go:122] "RemoveContainer" containerID="99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"
Oct 30 22:17:21.814876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.814613    4295 scope.go:122] "RemoveContainer" containerID="eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"
Oct 30 22:17:21.815066 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:21.815036    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863\": container with ID starting with eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863 not found: ID does not exist" containerID="eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"
Oct 30 22:17:21.815167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.815079    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863"} err="failed to get container status \"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863\": rpc error: code = NotFound desc = could not find container \"eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863\": container with ID starting with eb7065d411f889a734d064eb2fbe06fb9262a575056ecd1f3cdc0fc4e4251863 not found: ID does not exist"
Oct 30 22:17:21.815167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.815106    4295 scope.go:122] "RemoveContainer" containerID="99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"
Oct 30 22:17:21.815558 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:21.815438    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e\": container with ID starting with 99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e not found: ID does not exist" containerID="99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"
Oct 30 22:17:21.815686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:21.815570    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e"} err="failed to get container status \"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e\": rpc error: code = NotFound desc = could not find container \"99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e\": container with ID starting with 99bca2de1428aae387eafbc6be69a75716b3695c88420681461e28821624ad7e not found: ID does not exist"
Oct 30 22:17:22.052234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:22.052134    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:17:22.096690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:22.077325    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d"]
Oct 30 22:17:23.494603 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:23.494571    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" path="/var/lib/kubelet/pods/544e6d2e-ba0f-4029-b7c8-a45fe75e2770/volumes"
Oct 30 22:17:23.609051 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:23.609019    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:17:34.101842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.101805    4295 generic.go:356] "Generic (PLEG): container finished" podID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerID="fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91" exitCode=0
Oct 30 22:17:34.102365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.101881    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerDied","Data":"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"}
Oct 30 22:17:34.102365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.102334    4295 scope.go:122] "RemoveContainer" containerID="fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"
Oct 30 22:17:34.104028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.103999    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:34.104107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.104078    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:34.104145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:34.104112    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:35.126212 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:35.126182    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerStarted","Data":"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"}
Oct 30 22:17:35.841723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:35.841658    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:17:36.128735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:36.120749    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:17:36.128735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:36.121055    4295 patch_prober.go:29] interesting pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:17:36.128735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:36.121133    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:17:37.113829 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:37.113791    4295 patch_prober.go:29] interesting pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:17:37.114118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:37.114088    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:17:38.122098 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:38.122058    4295 patch_prober.go:29] interesting pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:17:38.122754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:38.122714    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:17:39.113118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:39.113016    4295 patch_prober.go:29] interesting pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:17:39.113118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:39.113068    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:17:40.113377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:40.113335    4295 patch_prober.go:29] interesting pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:17:40.113932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:40.113787    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:17:41.116991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:41.116962    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:17:41.118020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:17:41.118001    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:17:53.609453 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:17:53.609414    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:18:09.358832 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:09.358783    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:18:09.359268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:09.358847    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:10.870488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.870454    4295 generic.go:356] "Generic (PLEG): container finished" podID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerID="55fe89166a1afe1ca22857b31673892c6a64231e72d7199dbd77444efa983f60" exitCode=0
Oct 30 22:18:10.870954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.870525    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerDied","Data":"55fe89166a1afe1ca22857b31673892c6a64231e72d7199dbd77444efa983f60"}
Oct 30 22:18:10.870954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.870945    4295 scope.go:122] "RemoveContainer" containerID="55fe89166a1afe1ca22857b31673892c6a64231e72d7199dbd77444efa983f60"
Oct 30 22:18:10.872392 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.872356    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:10.872487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.872438    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:10.872487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:10.872476    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:11.964688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:11.964041    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerStarted","Data":"8b6c2af9002394893f76e0677b97ed277020d7c5cf859426ca7431ea67f2210e"}
Oct 30 22:18:12.563688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:12.556586    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:18:12.630489 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:12.630455    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:18:12.638649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:12.638608    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 22:18:13.470653 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:13.470615    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:18:13.471062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:13.470706    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 22:18:14.470147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:14.470109    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:18:14.470326 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:14.470171    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 22:18:15.470201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:15.470163    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:18:15.470629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:15.470222    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 22:18:16.470756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:16.470718    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:18:16.471247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:16.470784    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 22:18:17.475238 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:17.475201    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:18:17.475731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:17.475585    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:18:17.476342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:17.476323    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:18:20.937018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.936981    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:18:20.939817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.939789    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy"
Oct 30 22:18:20.939990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.939976    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy"
Oct 30 22:18:20.940089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940079    4295 cpu_manager.go:414] "containerMap: removing container" podUID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" containerName="maas-api"
Oct 30 22:18:20.940167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940156    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" containerName="maas-api"
Oct 30 22:18:20.940273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940262    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy"
Oct 30 22:18:20.940351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940339    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy"
Oct 30 22:18:20.940470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940461    4295 cpu_manager.go:414] "containerMap: removing container" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.940547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940536    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.940629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940618    4295 cpu_manager.go:414] "containerMap: removing container" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.940729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940717    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.940817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940807    4295 cpu_manager.go:414] "containerMap: removing container" podUID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" containerName="maas-api"
Oct 30 22:18:20.940890 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.940880    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" containerName="maas-api"
Oct 30 22:18:20.941145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941131    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1733c0cf-3922-48fb-aa68-0c9ffbeb329e" containerName="istio-proxy"
Oct 30 22:18:20.941250 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941240    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="df00b75d-4ed8-4ad2-abc4-da8f0a309f83" containerName="maas-api"
Oct 30 22:18:20.941333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941324    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f971d96b-f399-4825-8dfa-83a581ece4dc" containerName="istio-proxy"
Oct 30 22:18:20.941416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941407    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.941491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941481    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="544e6d2e-ba0f-4029-b7c8-a45fe75e2770" containerName="istio-proxy"
Oct 30 22:18:20.941562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.941554    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c68ef0-ef33-4195-b1c2-93c3e67107cf" containerName="maas-api"
Oct 30 22:18:20.945126 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:20.945103    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.066224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.066190    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c3d917-dockercfg-mglgr\""
Oct 30 22:18:21.067154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.067128    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c3d917\""
Oct 30 22:18:21.102697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.088034    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.102697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.088266    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687sk\" (UniqueName: \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.102697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.091762    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:18:21.205960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.205856    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-687sk\" (UniqueName: \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.206147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.206051    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.207018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.206990    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.277255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.277222    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-687sk\" (UniqueName: \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\") pod \"payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.349638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.349608    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:21.932162 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:18:21.932131    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d75289_e598_4146_85f2_c64ae0b3c60f.slice/crio-566a417e33b9910238494d872ed070632b0d1137eecd45fbd218545e93c8963f WatchSource:0}: Error finding container 566a417e33b9910238494d872ed070632b0d1137eecd45fbd218545e93c8963f: Status 404 returned error can't find the container with id 566a417e33b9910238494d872ed070632b0d1137eecd45fbd218545e93c8963f
Oct 30 22:18:21.932298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:21.932173    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:18:22.101140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:22.101107    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" event={"ID":"44d75289-e598-4146-85f2-c64ae0b3c60f","Type":"ContainerStarted","Data":"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"}
Oct 30 22:18:22.101555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:22.101166    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" event={"ID":"44d75289-e598-4146-85f2-c64ae0b3c60f","Type":"ContainerStarted","Data":"566a417e33b9910238494d872ed070632b0d1137eecd45fbd218545e93c8963f"}
Oct 30 22:18:22.101555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:22.101195    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:22.223972 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:22.223856    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" podStartSLOduration=2.223841513 podStartE2EDuration="2.223841513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:22.220272241 +0000 UTC m=+1704.862866028" watchObservedRunningTime="2026-10-30 22:18:22.223841513 +0000 UTC m=+1704.866435302"
Oct 30 22:18:23.608887 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:23.608849    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:18:33.134097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:33.134067    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:18:36.354606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.354576    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:18:36.355074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.354805    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" containerID="cri-o://ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2" gracePeriod=30
Oct 30 22:18:36.404798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.404760    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:18:36.405017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.404983    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" containerID="cri-o://e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8" gracePeriod=30
Oct 30 22:18:36.407894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.407870    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:18:36.408071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.408053    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" containerID="cri-o://444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726" gracePeriod=30
Oct 30 22:18:36.456158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.456124    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:18:36.456413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:36.456359    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" containerID="cri-o://ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76" gracePeriod=30
Oct 30 22:18:37.695997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:37.695953    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:37.696414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:37.696035    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:38.853396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:38.853361    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:38.853895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:38.853444    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:39.798677 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:39.798627    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:39.798912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:39.798701    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:40.213692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:40.213108    4295 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:40.213692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:40.213187    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:41.711562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.711535    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:18:41.766765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.766719    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:18:41.771591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.771564    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:18:41.813405 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813382    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813476    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813536    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.813576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813556    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813605    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.813707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813630    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813685    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.813877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813715    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813750    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813787    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.813877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813825    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.813877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.813861    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814045 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814001    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814042    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.814089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814067    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.814166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814080    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.814166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814086    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814157    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.814273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814174    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814243    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814281    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\") pod \"bd25fa86-1783-4a60-a3a9-60c2e858412c\" (UID: \"bd25fa86-1783-4a60-a3a9-60c2e858412c\") "
Oct 30 22:18:41.814416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814330    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\") pod \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\" (UID: \"b46d198d-51f4-4a97-8135-74d9b02ebf2c\") "
Oct 30 22:18:41.814760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814732    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.814867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814762    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.814936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.814915    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.815176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815150    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.815337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815315    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815344    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815350    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815360    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd25fa86-1783-4a60-a3a9-60c2e858412c-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815375    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815391    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815405    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815421    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815437    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.815719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.815530    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.816401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.816351    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.817139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.817114    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.817234 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.817186    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.818116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.818093    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "kube-api-access-7rz6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.818272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.818256    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:41.818544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.818518    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh" pod "b46d198d-51f4-4a97-8135-74d9b02ebf2c" (UID: "b46d198d-51f4-4a97-8135-74d9b02ebf2c"). InnerVolumeSpecName "kube-api-access-mv7fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.818939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.818910    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:41.819175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.819147    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy" pod "bd25fa86-1783-4a60-a3a9-60c2e858412c" (UID: "bd25fa86-1783-4a60-a3a9-60c2e858412c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.821407 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.821386    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:18:41.916733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916633    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.916733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916720    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.916959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916769    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.916959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916810    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.916959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916847    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.916959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916902    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.916959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916930    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.916983    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.917209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917016    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917094    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917008    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917177    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917189    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917222    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917290    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917350 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917332    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917377    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917428    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917436    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917465    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\") pod \"db4d64d0-1ec7-4416-a917-01735590a78c\" (UID: \"db4d64d0-1ec7-4416-a917-01735590a78c\") "
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917535    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.917687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917601    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\") pod \"7d6568c8-6bbe-49d8-8acc-c267104ae995\" (UID: \"7d6568c8-6bbe-49d8-8acc-c267104ae995\") "
Oct 30 22:18:41.918170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.917868    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.918231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918211    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.918477 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918392    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.918477 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918453    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.918477 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918468    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918487    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918501    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7fh\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-kube-api-access-mv7fh\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918513    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918525    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918538    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918551    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918566    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918578    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918575    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918590    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4d64d0-1ec7-4416-a917-01735590a78c-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918604    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918619    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918633    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rz6m\" (UniqueName: \"kubernetes.io/projected/bd25fa86-1783-4a60-a3a9-60c2e858412c-kube-api-access-7rz6m\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918648    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d198d-51f4-4a97-8135-74d9b02ebf2c-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918699    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.918740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.918715    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd25fa86-1783-4a60-a3a9-60c2e858412c-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.919914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.919197    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.920128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920008    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:41.920128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920023    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.920385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920358    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:41.920520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920446    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.920590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920525    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:41.920936 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.920912    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l" pod "7d6568c8-6bbe-49d8-8acc-c267104ae995" (UID: "7d6568c8-6bbe-49d8-8acc-c267104ae995"). InnerVolumeSpecName "kube-api-access-rwm7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.921194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.921176    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "kube-api-access-7w8xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.921468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:41.921453    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token" pod "db4d64d0-1ec7-4416-a917-01735590a78c" (UID: "db4d64d0-1ec7-4416-a917-01735590a78c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:42.020277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020229    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020261    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020271    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020280    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w8xr\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-kube-api-access-7w8xr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020291    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4d64d0-1ec7-4416-a917-01735590a78c-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020300    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4d64d0-1ec7-4416-a917-01735590a78c-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020309    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020317    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020325    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6568c8-6bbe-49d8-8acc-c267104ae995-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020333    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6568c8-6bbe-49d8-8acc-c267104ae995-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020341    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4d64d0-1ec7-4416-a917-01735590a78c-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.020630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.020350    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm7l\" (UniqueName: \"kubernetes.io/projected/7d6568c8-6bbe-49d8-8acc-c267104ae995-kube-api-access-rwm7l\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.288932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.288901    4295 generic.go:356] "Generic (PLEG): container finished" podID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerID="ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2" exitCode=0
Oct 30 22:18:42.289132 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.288955    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"
Oct 30 22:18:42.289132 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.288986    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" event={"ID":"bd25fa86-1783-4a60-a3a9-60c2e858412c","Type":"ContainerDied","Data":"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"}
Oct 30 22:18:42.289132 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.289084    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn" event={"ID":"bd25fa86-1783-4a60-a3a9-60c2e858412c","Type":"ContainerDied","Data":"98369256faaf80a42c6c9f6c961e8ea125d0318b8859aa9b9f616786de2ac62d"}
Oct 30 22:18:42.289132 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.289114    4295 scope.go:122] "RemoveContainer" containerID="ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"
Oct 30 22:18:42.290508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.290477    4295 generic.go:356] "Generic (PLEG): container finished" podID="db4d64d0-1ec7-4416-a917-01735590a78c" containerID="444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726" exitCode=0
Oct 30 22:18:42.290616 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.290510    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" event={"ID":"db4d64d0-1ec7-4416-a917-01735590a78c","Type":"ContainerDied","Data":"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"}
Oct 30 22:18:42.290616 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.290566    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"
Oct 30 22:18:42.290616 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.290572    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr" event={"ID":"db4d64d0-1ec7-4416-a917-01735590a78c","Type":"ContainerDied","Data":"b678fd72263f632637b076450a2519c9b615a1511e3e9a452ce888aac973c2cc"}
Oct 30 22:18:42.292136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.292098    4295 generic.go:356] "Generic (PLEG): container finished" podID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerID="ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76" exitCode=0
Oct 30 22:18:42.292269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.292159    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"
Oct 30 22:18:42.292269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.292178    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" event={"ID":"7d6568c8-6bbe-49d8-8acc-c267104ae995","Type":"ContainerDied","Data":"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"}
Oct 30 22:18:42.292269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.292225    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5" event={"ID":"7d6568c8-6bbe-49d8-8acc-c267104ae995","Type":"ContainerDied","Data":"4403f44878acf0a90eb80f3f35c545377dd8bb1a4104a398d7990674aa8fa527"}
Oct 30 22:18:42.298895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.298870    4295 generic.go:356] "Generic (PLEG): container finished" podID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerID="e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8" exitCode=0
Oct 30 22:18:42.299067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.298930    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"
Oct 30 22:18:42.299067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.298948    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" event={"ID":"b46d198d-51f4-4a97-8135-74d9b02ebf2c","Type":"ContainerDied","Data":"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"}
Oct 30 22:18:42.299067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.299004    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x" event={"ID":"b46d198d-51f4-4a97-8135-74d9b02ebf2c","Type":"ContainerDied","Data":"1aabf3f3e6eca58357fb3247b70ea5f140e1885a1e281f7e9309dd4656057a2f"}
Oct 30 22:18:42.302410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.302393    4295 scope.go:122] "RemoveContainer" containerID="ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"
Oct 30 22:18:42.302761 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:42.302741    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2\": container with ID starting with ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2 not found: ID does not exist" containerID="ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"
Oct 30 22:18:42.302853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.302767    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2"} err="failed to get container status \"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2\": rpc error: code = NotFound desc = could not find container \"ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2\": container with ID starting with ab9c4c2eb126faccfc6b64862d830e7b0c5191ae432d0a43255620b8adbb13e2 not found: ID does not exist"
Oct 30 22:18:42.302853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.302790    4295 scope.go:122] "RemoveContainer" containerID="444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"
Oct 30 22:18:42.315208 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.315191    4295 scope.go:122] "RemoveContainer" containerID="444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"
Oct 30 22:18:42.315502 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:42.315483    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726\": container with ID starting with 444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726 not found: ID does not exist" containerID="444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"
Oct 30 22:18:42.315581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.315511    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726"} err="failed to get container status \"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726\": rpc error: code = NotFound desc = could not find container \"444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726\": container with ID starting with 444d46856cf605475a332443166c372b1cf8653ab2163c2efa2fe930342c0726 not found: ID does not exist"
Oct 30 22:18:42.315581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.315534    4295 scope.go:122] "RemoveContainer" containerID="ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"
Oct 30 22:18:42.325973 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.325958    4295 scope.go:122] "RemoveContainer" containerID="ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"
Oct 30 22:18:42.326212 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:42.326193    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76\": container with ID starting with ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76 not found: ID does not exist" containerID="ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"
Oct 30 22:18:42.326258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.326219    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76"} err="failed to get container status \"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76\": rpc error: code = NotFound desc = could not find container \"ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76\": container with ID starting with ad28a4154cc71b7fcb751cb1f964067f61cf001bed18953eae3e82ed53a23e76 not found: ID does not exist"
Oct 30 22:18:42.326258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.326234    4295 scope.go:122] "RemoveContainer" containerID="e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"
Oct 30 22:18:42.334934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.334917    4295 scope.go:122] "RemoveContainer" containerID="e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"
Oct 30 22:18:42.335199 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:42.335179    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8\": container with ID starting with e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8 not found: ID does not exist" containerID="e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"
Oct 30 22:18:42.335243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.335205    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8"} err="failed to get container status \"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8\": rpc error: code = NotFound desc = could not find container \"e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8\": container with ID starting with e5678932451a9682cc62c7cb3cda2af08f5c8804b5647ea65fca96db9fc3c6f8 not found: ID does not exist"
Oct 30 22:18:42.509627 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.509556    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:18:42.529238 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.529205    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x"]
Oct 30 22:18:42.597719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.597630    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:18:42.606558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.606533    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5"]
Oct 30 22:18:42.730907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.730864    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:18:42.734025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.733998    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn"]
Oct 30 22:18:42.846703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.846673    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:18:42.853301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:42.853221    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr"]
Oct 30 22:18:43.495232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:43.495202    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" path="/var/lib/kubelet/pods/7d6568c8-6bbe-49d8-8acc-c267104ae995/volumes"
Oct 30 22:18:43.495753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:43.495738    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" path="/var/lib/kubelet/pods/b46d198d-51f4-4a97-8135-74d9b02ebf2c/volumes"
Oct 30 22:18:43.496185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:43.496172    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" path="/var/lib/kubelet/pods/bd25fa86-1783-4a60-a3a9-60c2e858412c/volumes"
Oct 30 22:18:43.496965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:43.496950    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" path="/var/lib/kubelet/pods/db4d64d0-1ec7-4416-a917-01735590a78c/volumes"
Oct 30 22:18:46.367814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.367780    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:18:46.368410 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368394    4295 cpu_manager.go:414] "containerMap: removing container" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy"
Oct 30 22:18:46.368450 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368412    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368449    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368455    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368467    4295 cpu_manager.go:414] "containerMap: removing container" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368472    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368484    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy"
Oct 30 22:18:46.368494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368489    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy"
Oct 30 22:18:46.368674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368597    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4d64d0-1ec7-4416-a917-01735590a78c" containerName="istio-proxy"
Oct 30 22:18:46.368674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368610    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46d198d-51f4-4a97-8135-74d9b02ebf2c" containerName="istio-proxy"
Oct 30 22:18:46.368674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368618    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6568c8-6bbe-49d8-8acc-c267104ae995" containerName="istio-proxy"
Oct 30 22:18:46.368674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.368625    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd25fa86-1783-4a60-a3a9-60c2e858412c" containerName="istio-proxy"
Oct 30 22:18:46.372104 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.372087    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.433901 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.433874    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:18:46.481779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.481749    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-422c1029\"/\"openshift-service-ca.crt\""
Oct 30 22:18:46.483925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.482319    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-422c1029\"/\"default-dockercfg-r9qjn\""
Oct 30 22:18:46.484161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.483086    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-422c1029\"/\"e2e-lifecycle-model-422c1029-kserve-self-signed-certs\""
Oct 30 22:18:46.484275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.483187    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-422c1029\"/\"kube-root-ca.crt\""
Oct 30 22:18:46.507091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507046    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.507293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507103    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.507293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507180    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.507293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507233    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.507491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507313    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.507491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.507391    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cwp\" (UniqueName: \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.584707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.581685    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:18:46.588914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.588887    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.608328 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608297    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cwp\" (UniqueName: \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608456    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608499    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608599    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608686    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608796    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.608991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.608968    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.609182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.609157    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.609457 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.609433    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.611788 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.611766    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.612056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.612035    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.650097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.649922    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-512caa-serving-cert\""
Oct 30 22:18:46.650247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.650179    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:18:46.677414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.677379    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cwp\" (UniqueName: \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\") pod \"e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") " pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.687470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.687386    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:46.710524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.710290    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.711159 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.711120    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.711268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.711169    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.711268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.711208    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pq8\" (UniqueName: \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.813054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.813014    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pq8\" (UniqueName: \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.813222 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.813200    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.813430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.813405    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.813520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.813450    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.813630 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:46.813611    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-serving-cert: secret "maas-api-e2e-api-a-512caa-serving-cert" not found
Oct 30 22:18:46.813748 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:46.813701    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls podName:fa0d043d-2dc4-487b-a080-7c21f9183b84 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:47.313678641 +0000 UTC m=+1729.956272425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls") pod "maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" (UID: "fa0d043d-2dc4-487b-a080-7c21f9183b84") : secret "maas-api-e2e-api-a-512caa-serving-cert" not found
Oct 30 22:18:46.814725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.814700    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.816795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.816768    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:46.943701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:46.941841    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pq8\" (UniqueName: \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:47.165481 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.164421    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:18:47.169179 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:18:47.168862    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f499cce_8561_4ff6_b2f7_dd83e9dc00f7.slice/crio-29ae1b5fc3c4c86c29384c533166ff69a9f4c591580e4ecdfad95d426b6b9d4d WatchSource:0}: Error finding container 29ae1b5fc3c4c86c29384c533166ff69a9f4c591580e4ecdfad95d426b6b9d4d: Status 404 returned error can't find the container with id 29ae1b5fc3c4c86c29384c533166ff69a9f4c591580e4ecdfad95d426b6b9d4d
Oct 30 22:18:47.335274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.335244    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" event={"ID":"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7","Type":"ContainerStarted","Data":"bd96da5691711ae6bada8ec71333f11258bbd6a7404f2afe5a1ae2712706cae8"}
Oct 30 22:18:47.335421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.335289    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" event={"ID":"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7","Type":"ContainerStarted","Data":"29ae1b5fc3c4c86c29384c533166ff69a9f4c591580e4ecdfad95d426b6b9d4d"}
Oct 30 22:18:47.361306 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.361273    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:47.363819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.363782    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") pod \"maas-api-e2e-api-a-512caa-995fbbbc4-wqp82\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:47.463036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.462933    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" podStartSLOduration=1.462919686 podStartE2EDuration="1.462919686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:47.45831403 +0000 UTC m=+1730.100907816" watchObservedRunningTime="2026-10-30 22:18:47.462919686 +0000 UTC m=+1730.105513478"
Oct 30 22:18:47.504790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.504760    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:47.687976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.687909    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:47.981267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:47.965754    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:18:48.080406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:48.079406    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:18:48.343226 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:48.343134    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" event={"ID":"fa0d043d-2dc4-487b-a080-7c21f9183b84","Type":"ContainerStarted","Data":"ac686aafc7b4f4869e3d0168f88a312383a83344e2c5477d5e715f86a01a2b9b"}
Oct 30 22:18:49.352209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:49.352113    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" event={"ID":"fa0d043d-2dc4-487b-a080-7c21f9183b84","Type":"ContainerStarted","Data":"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"}
Oct 30 22:18:49.352631 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:49.352482    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:49.491042 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:49.490977    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" podStartSLOduration=2.634888249 podStartE2EDuration="3.490956036s" totalImagesPullingTime="856.067787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:49.482570302 +0000 UTC m=+1732.125164169" watchObservedRunningTime="2026-10-30 22:18:49.490956036 +0000 UTC m=+1732.133549826"
Oct 30 22:18:53.611336 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:18:53.611293    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:18:55.364118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:55.364090    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:18:56.688558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:56.688530    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:56.700986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:56.700954    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:18:57.423558 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:18:57.423529    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:19:01.104561 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:01.104525    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:19:01.223215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:01.223182    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:19:06.105049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.105003    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" podUID="55c46161-3737-41c4-914e-58cb1682ba14" containerName="payload-processing" containerID="cri-o://3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb" gracePeriod=25
Oct 30 22:19:06.223927 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.223888    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" podUID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" containerName="payload-pre-processing" containerID="cri-o://10a00668241cfcf43488330b636dbb51a1c5977d0852e1ceff85eba2d368b6a3" gracePeriod=25
Oct 30 22:19:06.385296 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.385269    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:19:06.473800 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.473760    4295 generic.go:356] "Generic (PLEG): container finished" podID="55c46161-3737-41c4-914e-58cb1682ba14" containerID="3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb" exitCode=0
Oct 30 22:19:06.474046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.473948    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" event={"ID":"55c46161-3737-41c4-914e-58cb1682ba14","Type":"ContainerDied","Data":"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"}
Oct 30 22:19:06.474046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.474001    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc" event={"ID":"55c46161-3737-41c4-914e-58cb1682ba14","Type":"ContainerDied","Data":"3c8bfca4f372c7c82c3b471e3316ea202583158127c737c7ea15a931f5e2c005"}
Oct 30 22:19:06.474046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.474031    4295 scope.go:122] "RemoveContainer" containerID="3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"
Oct 30 22:19:06.474241 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.474182    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"
Oct 30 22:19:06.476863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.476838    4295 generic.go:356] "Generic (PLEG): container finished" podID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" containerID="10a00668241cfcf43488330b636dbb51a1c5977d0852e1ceff85eba2d368b6a3" exitCode=0
Oct 30 22:19:06.477135 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.476916    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" event={"ID":"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8","Type":"ContainerDied","Data":"10a00668241cfcf43488330b636dbb51a1c5977d0852e1ceff85eba2d368b6a3"}
Oct 30 22:19:06.488900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.488881    4295 scope.go:122] "RemoveContainer" containerID="3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"
Oct 30 22:19:06.489237 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:06.489213    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb\": container with ID starting with 3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb not found: ID does not exist" containerID="3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"
Oct 30 22:19:06.489316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.489247    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb"} err="failed to get container status \"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb\": rpc error: code = NotFound desc = could not find container \"3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb\": container with ID starting with 3c44eb835e791e454e9c651c72816a0702715a6d97ce3238f479c1155bdcdebb not found: ID does not exist"
Oct 30 22:19:06.515479 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.515456    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:19:06.568872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.568833    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\" (UniqueName: \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\") pod \"55c46161-3737-41c4-914e-58cb1682ba14\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") "
Oct 30 22:19:06.569061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.568906    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\") pod \"55c46161-3737-41c4-914e-58cb1682ba14\" (UID: \"55c46161-3737-41c4-914e-58cb1682ba14\") "
Oct 30 22:19:06.569339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.569314    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume" pod "55c46161-3737-41c4-914e-58cb1682ba14" (UID: "55c46161-3737-41c4-914e-58cb1682ba14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:06.569456 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.569440    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c46161-3737-41c4-914e-58cb1682ba14-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.571380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.571350    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv" pod "55c46161-3737-41c4-914e-58cb1682ba14" (UID: "55c46161-3737-41c4-914e-58cb1682ba14"). InnerVolumeSpecName "kube-api-access-qgwrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:06.670548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.670474    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\" (UniqueName: \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\") pod \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") "
Oct 30 22:19:06.670708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.670696    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\") pod \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\" (UID: \"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8\") "
Oct 30 22:19:06.671034 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.671009    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume" pod "e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" (UID: "e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:06.671224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.671177    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwrv\" (UniqueName: \"kubernetes.io/projected/55c46161-3737-41c4-914e-58cb1682ba14-kube-api-access-qgwrv\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.671224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.671191    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.672815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.672789    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx" pod "e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" (UID: "e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8"). InnerVolumeSpecName "kube-api-access-ztphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:06.772361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.772320    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztphx\" (UniqueName: \"kubernetes.io/projected/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8-kube-api-access-ztphx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.904592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.904558    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:19:06.908965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:06.908939    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc"]
Oct 30 22:19:07.485177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.485152    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"
Oct 30 22:19:07.485691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.485193    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd" event={"ID":"e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8","Type":"ContainerDied","Data":"d7e739188f3d69e33749919c70ea4719d44d39ee5f92a6cb0af89e21723d74a7"}
Oct 30 22:19:07.485691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.485258    4295 scope.go:122] "RemoveContainer" containerID="10a00668241cfcf43488330b636dbb51a1c5977d0852e1ceff85eba2d368b6a3"
Oct 30 22:19:07.495702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.495655    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c46161-3737-41c4-914e-58cb1682ba14" path="/var/lib/kubelet/pods/55c46161-3737-41c4-914e-58cb1682ba14/volumes"
Oct 30 22:19:07.676810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.676781    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:19:07.729385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:07.729357    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd"]
Oct 30 22:19:09.512708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:09.512436    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" path="/var/lib/kubelet/pods/e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8/volumes"
Oct 30 22:19:19.406615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.406582    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:19:19.407188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407173    4295 cpu_manager.go:414] "containerMap: removing container" podUID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" containerName="payload-pre-processing"
Oct 30 22:19:19.407232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407191    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" containerName="payload-pre-processing"
Oct 30 22:19:19.407232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407218    4295 cpu_manager.go:414] "containerMap: removing container" podUID="55c46161-3737-41c4-914e-58cb1682ba14" containerName="payload-processing"
Oct 30 22:19:19.407232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407223    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c46161-3737-41c4-914e-58cb1682ba14" containerName="payload-processing"
Oct 30 22:19:19.407327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407319    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f873a6-2c04-4e13-a5e8-f5ff41fdb4a8" containerName="payload-pre-processing"
Oct 30 22:19:19.407364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.407331    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c46161-3737-41c4-914e-58cb1682ba14" containerName="payload-processing"
Oct 30 22:19:19.410577 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.410563    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.471363 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.471337    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:19:19.517303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.517261    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9b614c-serving-cert\""
Oct 30 22:19:19.550746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.550708    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.550908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.550834    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspxt\" (UniqueName: \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.550908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.550896    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.550982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.550913    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.654777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.654730    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.695846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.690946    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspxt\" (UniqueName: \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.695846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.691058    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.695846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.691088    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.695846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.668048    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.712639 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.712605    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.713056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.712960    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:19.755221 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:19.755183    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspxt\" (UniqueName: \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\") pod \"maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:20.024963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:20.024881    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:20.408038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:20.408006    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:19:20.410009 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:19:20.409982    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121a83ce_3115_40e6_8e13_984f099c6204.slice/crio-28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d WatchSource:0}: Error finding container 28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d: Status 404 returned error can't find the container with id 28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d
Oct 30 22:19:20.645086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:20.645045    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" event={"ID":"121a83ce-3115-40e6-8e13-984f099c6204","Type":"ContainerStarted","Data":"28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d"}
Oct 30 22:19:21.694704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:21.691102    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" event={"ID":"121a83ce-3115-40e6-8e13-984f099c6204","Type":"ContainerStarted","Data":"d2b23d56e6ea95d8ca2baf07ee0bc57db5ad3f806dfe4531f0c37a4889f27907"}
Oct 30 22:19:21.694704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:21.691181    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:21.815837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:21.815787    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" podStartSLOduration=1.976900656 podStartE2EDuration="2.815774171s" totalImagesPullingTime="838.873515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:21.813809112 +0000 UTC m=+1764.456402901" watchObservedRunningTime="2026-10-30 22:19:21.815774171 +0000 UTC m=+1764.458367959"
Oct 30 22:19:23.610154 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:23.610120    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:19:27.728387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:27.728354    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:19:34.401258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.401218    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:19:34.401728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.401441    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" podUID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" containerName="maas-api" containerID="cri-o://11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914" gracePeriod=30
Oct 30 22:19:34.780562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.780538    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:19:34.852695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.834227    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\") pod \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") "
Oct 30 22:19:34.852695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.834389    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\" (UniqueName: \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\") pod \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") "
Oct 30 22:19:34.852695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.834457    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") pod \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") "
Oct 30 22:19:34.852695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.834553    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\") pod \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\" (UID: \"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254\") "
Oct 30 22:19:34.861715 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.857241    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca" pod "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" (UID: "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:34.931831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.931564    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf" pod "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" (UID: "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254"). InnerVolumeSpecName "kube-api-access-p78hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:34.931831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.931744    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs" pod "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" (UID: "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:34.931831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.931801    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls" pod "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" (UID: "7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:34.944730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.942675    4295 generic.go:356] "Generic (PLEG): container finished" podID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" containerID="11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914" exitCode=0
Oct 30 22:19:34.944730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.942866    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" event={"ID":"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254","Type":"ContainerDied","Data":"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"}
Oct 30 22:19:34.944730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.942943    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh" event={"ID":"7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254","Type":"ContainerDied","Data":"08feb071bb00d3bd3121f0f452ee2a0fb90b6b8d471bd4443a8c8f1bfd3bf00a"}
Oct 30 22:19:34.944730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.942981    4295 scope.go:122] "RemoveContainer" containerID="11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"
Oct 30 22:19:34.944730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.943171    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"
Oct 30 22:19:34.952004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.951973    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.952257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.952238    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78hf\" (UniqueName: \"kubernetes.io/projected/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-kube-api-access-p78hf\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.952372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.952360    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.952447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:34.952437    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.005907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:35.005883    4295 scope.go:122] "RemoveContainer" containerID="11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"
Oct 30 22:19:35.006344 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:35.006307    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914\": container with ID starting with 11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914 not found: ID does not exist" containerID="11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"
Oct 30 22:19:35.006459 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:35.006355    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914"} err="failed to get container status \"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914\": rpc error: code = NotFound desc = could not find container \"11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914\": container with ID starting with 11dd6bad77390c3fb4bdb496b45df1fab3dd4a7694bc563f225d76c9bee9c914 not found: ID does not exist"
Oct 30 22:19:35.137468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:35.137433    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:19:35.141816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:35.141788    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7422b4b3-76fdb84c7b-fzrhh"]
Oct 30 22:19:35.542688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:35.536000    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" path="/var/lib/kubelet/pods/7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254/volumes"
Oct 30 22:19:36.041213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.040715    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:19:36.041566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.041545    4295 cpu_manager.go:414] "containerMap: removing container" podUID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" containerName="maas-api"
Oct 30 22:19:36.041566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.041566    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" containerName="maas-api"
Oct 30 22:19:36.041702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.041697    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2cd0af-2ab1-4b60-8cf7-47e3e4e21254" containerName="maas-api"
Oct 30 22:19:36.046341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.046325    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.074640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.074614    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.074764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.074679    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.074807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.074766    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47q7\" (UniqueName: \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.074984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.074962    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.102978 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.102947    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c3d917-serving-cert\""
Oct 30 22:19:36.108217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.108178    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:19:36.176600 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.176563    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.176824 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.176698    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.176824 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.176740    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47q7\" (UniqueName: \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.182701 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:36.176846    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-serving-cert: secret "maas-api-e2e-shared-b-w0-c3d917-serving-cert" not found
Oct 30 22:19:36.182852 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:36.182755    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls podName:18390268-bd86-4414-953e-be17364ef40f nodeName:}" failed. No retries permitted until 2026-10-30 22:19:36.68273745 +0000 UTC m=+1779.325331230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" (UID: "18390268-bd86-4414-953e-be17364ef40f") : secret "maas-api-e2e-shared-b-w0-c3d917-serving-cert" not found
Oct 30 22:19:36.182852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.177214    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.182852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.179762    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.182852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.176968    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.240561 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.240529    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47q7\" (UniqueName: \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.698405 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.698370    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.700977 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.700955    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:36.960125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:36.960049    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:37.341875 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:19:37.341831    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18390268_bd86_4414_953e_be17364ef40f.slice/crio-459c7ee135ff31668f337a83ebcad3b5692e6301d6759fb585984c4652ba2b9f WatchSource:0}: Error finding container 459c7ee135ff31668f337a83ebcad3b5692e6301d6759fb585984c4652ba2b9f: Status 404 returned error can't find the container with id 459c7ee135ff31668f337a83ebcad3b5692e6301d6759fb585984c4652ba2b9f
Oct 30 22:19:37.342389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:37.342360    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:19:37.465385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:37.465352    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:19:37.465573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:37.465548    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" containerID="cri-o://c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b" gracePeriod=30
Oct 30 22:19:37.963189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:37.963156    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" event={"ID":"18390268-bd86-4414-953e-be17364ef40f","Type":"ContainerStarted","Data":"459c7ee135ff31668f337a83ebcad3b5692e6301d6759fb585984c4652ba2b9f"}
Oct 30 22:19:38.971992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:38.971955    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" event={"ID":"18390268-bd86-4414-953e-be17364ef40f","Type":"ContainerStarted","Data":"e9944a096344a17c1adea7c45e1eb8703edb6a7415e624b72da053641a6e123d"}
Oct 30 22:19:38.972411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:38.972036    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:39.099937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:39.099882    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" podStartSLOduration=3.3053991480000002 podStartE2EDuration="4.099866216s" totalImagesPullingTime="794.467068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:39.098075386 +0000 UTC m=+1781.740669178" watchObservedRunningTime="2026-10-30 22:19:39.099866216 +0000 UTC m=+1781.742460005"
Oct 30 22:19:41.294031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:41.293991    4295 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:41.294405 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:41.294059    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:42.759008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.758983    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:19:42.849855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.849818    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.849914    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.849940    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.849974    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850026    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850046    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850122    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850228    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850311    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\") pod \"0f467322-1c70-4132-9a74-d755b3a1edb7\" (UID: \"0f467322-1c70-4132-9a74-d755b3a1edb7\") "
Oct 30 22:19:42.850461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850342    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.850461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850363    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.850461 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850419    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:42.850844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850656    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.850844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.850691    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.851200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.851173    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.851309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.851201    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.851309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.851219    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.851309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.851234    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.851309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.851248    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f467322-1c70-4132-9a74-d755b3a1edb7-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.852480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.852444    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:42.852594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.852533    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "kube-api-access-mppcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:42.852594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.852548    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:42.852711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.852652    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy" pod "0f467322-1c70-4132-9a74-d755b3a1edb7" (UID: "0f467322-1c70-4132-9a74-d755b3a1edb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.952900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.952865    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppcd\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-kube-api-access-mppcd\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.952900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.952897    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.953093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.952910    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.953093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:42.952928    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f467322-1c70-4132-9a74-d755b3a1edb7-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:43.012199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.012109    4295 generic.go:356] "Generic (PLEG): container finished" podID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerID="c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b" exitCode=0
Oct 30 22:19:43.012199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.012162    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"
Oct 30 22:19:43.012385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.012195    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" event={"ID":"0f467322-1c70-4132-9a74-d755b3a1edb7","Type":"ContainerDied","Data":"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"}
Oct 30 22:19:43.012385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.012254    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs" event={"ID":"0f467322-1c70-4132-9a74-d755b3a1edb7","Type":"ContainerDied","Data":"c74ee16b5bb690e7c166ea8c1074b3f9fff4faa0a0b604320bcfb3c7c4da911c"}
Oct 30 22:19:43.012385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.012276    4295 scope.go:122] "RemoveContainer" containerID="c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"
Oct 30 22:19:43.023195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.023176    4295 scope.go:122] "RemoveContainer" containerID="c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"
Oct 30 22:19:43.023434 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:43.023413    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b\": container with ID starting with c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b not found: ID does not exist" containerID="c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"
Oct 30 22:19:43.023487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.023450    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b"} err="failed to get container status \"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b\": rpc error: code = NotFound desc = could not find container \"c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b\": container with ID starting with c5ed27cdb91af73efe83907b49986cba01ddb3516491206070f30da04306647b not found: ID does not exist"
Oct 30 22:19:43.182650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.182615    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:19:43.188090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.188064    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs"]
Oct 30 22:19:43.495037 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:43.495002    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" path="/var/lib/kubelet/pods/0f467322-1c70-4132-9a74-d755b3a1edb7/volumes"
Oct 30 22:19:44.987258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:44.987232    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:19:45.330934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:45.330842    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:19:45.331107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:45.331037    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy" containerID="cri-o://06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8" gracePeriod=30
Oct 30 22:19:47.016689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.016586    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:19:47.020696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.017570    4295 cpu_manager.go:414] "containerMap: removing container" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy"
Oct 30 22:19:47.020696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.017594    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy"
Oct 30 22:19:47.040688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.037265    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f467322-1c70-4132-9a74-d755b3a1edb7" containerName="istio-proxy"
Oct 30 22:19:47.177935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.169185    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:19:47.177935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.169336    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.246537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.246505    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-81f37a-openshift-default-dockercfg-f4dxx\""
Oct 30 22:19:47.266909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.266797    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.267385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.267355    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.267593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.267576    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.267822 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.267804    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.268008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.267990    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqxx\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.268318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.268301    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.268483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.268467    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.268620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.268604    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.268819 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.268800    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370521    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370559    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370576    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370595    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370648    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370794    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370824    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.370891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370870    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.370892    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqxx\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.371009    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.371103    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.371133    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371287 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.371267    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.371598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.371579    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.373091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.373071    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.373579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.373556    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.463626 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.452371    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.508701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.499440    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqxx\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\") pod \"e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") " pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.521771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.519893    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:47.979722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.979687    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:19:47.980314 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:19:47.980289    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663f8c63_7a5e_4212_936d_7560f1daa7cd.slice/crio-443f5204968d262911be17cd267d132e08d076d2e63ee74ac9e14a14086762f9 WatchSource:0}: Error finding container 443f5204968d262911be17cd267d132e08d076d2e63ee74ac9e14a14086762f9: Status 404 returned error can't find the container with id 443f5204968d262911be17cd267d132e08d076d2e63ee74ac9e14a14086762f9
Oct 30 22:19:47.983169 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.983136    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:47.983238 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.983205    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:47.983238 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:47.983233    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:48.053838 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:48.053805    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerStarted","Data":"443f5204968d262911be17cd267d132e08d076d2e63ee74ac9e14a14086762f9"}
Oct 30 22:19:49.061492 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:49.061454    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerStarted","Data":"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"}
Oct 30 22:19:49.525176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:49.525141    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:49.531170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:49.531140    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:49.668041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:49.667982    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podStartSLOduration=3.667962923 podStartE2EDuration="3.667962923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:49.245327694 +0000 UTC m=+1791.887921484" watchObservedRunningTime="2026-10-30 22:19:49.667962923 +0000 UTC m=+1792.310556714"
Oct 30 22:19:50.067717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.067688    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:50.068548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.068532    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:19:50.647522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.647501    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:19:50.734603 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.733937    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.743617    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.743769    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.743887    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.743980    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.744048    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.744145    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.744204    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.744268    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.745649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.744305    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\") pod \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\" (UID: \"6bfb937b-688b-4487-9507-7dc1c1e0ac7e\") "
Oct 30 22:19:50.746406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.746057    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:50.746406 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.746112    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:50.750886 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.750835    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:50.751281 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.751251    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:50.754164 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.753933    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:50.754472 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.754437    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "kube-api-access-5fphl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:50.754744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.754721    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:50.756397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.756370    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token" pod "6bfb937b-688b-4487-9507-7dc1c1e0ac7e" (UID: "6bfb937b-688b-4487-9507-7dc1c1e0ac7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860141    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fphl\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-kube-api-access-5fphl\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860177    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860200    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860215    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860232    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860247    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860261    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860276    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:50.869740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:50.860289    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bfb937b-688b-4487-9507-7dc1c1e0ac7e-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:51.075756 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.075724    4295 generic.go:356] "Generic (PLEG): container finished" podID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerID="06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8" exitCode=0
Oct 30 22:19:51.076202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.075766    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"
Oct 30 22:19:51.076202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.075801    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerDied","Data":"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"}
Oct 30 22:19:51.076202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.075862    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w" event={"ID":"6bfb937b-688b-4487-9507-7dc1c1e0ac7e","Type":"ContainerDied","Data":"e64dcde7888b0793c56bc91ebff046011789815a8c63242ecb704c58ae9f339a"}
Oct 30 22:19:51.076202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.075885    4295 scope.go:122] "RemoveContainer" containerID="06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"
Oct 30 22:19:51.086699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.086649    4295 scope.go:122] "RemoveContainer" containerID="3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"
Oct 30 22:19:51.096287 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.096269    4295 scope.go:122] "RemoveContainer" containerID="06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"
Oct 30 22:19:51.096537 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:51.096517    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8\": container with ID starting with 06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8 not found: ID does not exist" containerID="06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"
Oct 30 22:19:51.096608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.096549    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8"} err="failed to get container status \"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8\": rpc error: code = NotFound desc = could not find container \"06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8\": container with ID starting with 06f95ac29772b25c4f05f968002cfadf03184336c5009a6e191b383ead57d2b8 not found: ID does not exist"
Oct 30 22:19:51.096608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.096576    4295 scope.go:122] "RemoveContainer" containerID="3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"
Oct 30 22:19:51.096849 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:51.096832    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9\": container with ID starting with 3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9 not found: ID does not exist" containerID="3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"
Oct 30 22:19:51.096893 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.096855    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9"} err="failed to get container status \"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9\": rpc error: code = NotFound desc = could not find container \"3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9\": container with ID starting with 3b7b74c750ab28bf0d5d24345b13b75283472ab43e12fa91bcd91d3121199dd9 not found: ID does not exist"
Oct 30 22:19:51.252440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.252402    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:19:51.256457 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.256430    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w"]
Oct 30 22:19:51.535566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:51.535535    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" path="/var/lib/kubelet/pods/6bfb937b-688b-4487-9507-7dc1c1e0ac7e/volumes"
Oct 30 22:19:53.609562 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:53.609528    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:19:53.886161 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:53.886073    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:19:53.886316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:53.886296    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" containerID="cri-o://7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68" gracePeriod=30
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.047517    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048485    4295 cpu_manager.go:414] "containerMap: removing container" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048507    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048537    4295 cpu_manager.go:414] "containerMap: removing container" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048545    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048749    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.048764    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfb937b-688b-4487-9507-7dc1c1e0ac7e" containerName="istio-proxy"
Oct 30 22:19:56.089728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.054319    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.096534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.096496    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:19:56.160761 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.160720    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-81f37a\""
Oct 30 22:19:56.160997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.160812    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-81f37a-dockercfg-nkchp\""
Oct 30 22:19:56.185452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.185425    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.185598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.185493    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfvx\" (UniqueName: \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.287429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.287288    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.287429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.287390    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfvx\" (UniqueName: \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.288199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.288173    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.489352 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.489316    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:19:56.504721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.489580    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy" containerID="cri-o://40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786" gracePeriod=30
Oct 30 22:19:56.550111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.531287    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfvx\" (UniqueName: \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\") pod \"payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.550111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.549043    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:56.955522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:56.955498    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:19:56.956582 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:19:56.956556    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc82963e_4ebd_4965_8919_861c2bd880b9.slice/crio-f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2 WatchSource:0}: Error finding container f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2: Status 404 returned error can't find the container with id f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2
Oct 30 22:19:57.158990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.158956    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" event={"ID":"bc82963e-4ebd-4965-8919-861c2bd880b9","Type":"ContainerStarted","Data":"cc78964f664741e068af0b0dab526b50b2086c79740bd41874a2c9c9542abfe0"}
Oct 30 22:19:57.159368 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.159013    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" event={"ID":"bc82963e-4ebd-4965-8919-861c2bd880b9","Type":"ContainerStarted","Data":"f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2"}
Oct 30 22:19:57.159368 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.159036    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:19:57.279165 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.279065    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" podStartSLOduration=2.279051092 podStartE2EDuration="2.279051092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:57.278592774 +0000 UTC m=+1799.921186564" watchObservedRunningTime="2026-10-30 22:19:57.279051092 +0000 UTC m=+1799.921644881"
Oct 30 22:19:57.673063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.672982    4295 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:57.673063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:57.673049    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:59.211494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.211470    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:19:59.249954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.249920    4295 generic.go:356] "Generic (PLEG): container finished" podID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerID="7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68" exitCode=0
Oct 30 22:19:59.250078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.249987    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"
Oct 30 22:19:59.250078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.250004    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" event={"ID":"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502","Type":"ContainerDied","Data":"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"}
Oct 30 22:19:59.250181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.250084    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67" event={"ID":"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502","Type":"ContainerDied","Data":"51de4a8468c352fec97e22d643cc5553479efd28f7b8b2f326c7c33d63b5202b"}
Oct 30 22:19:59.250181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.250106    4295 scope.go:122] "RemoveContainer" containerID="7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"
Oct 30 22:19:59.259979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.259963    4295 scope.go:122] "RemoveContainer" containerID="7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"
Oct 30 22:19:59.260247 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:19:59.260225    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68\": container with ID starting with 7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68 not found: ID does not exist" containerID="7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"
Oct 30 22:19:59.260301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.260258    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68"} err="failed to get container status \"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68\": rpc error: code = NotFound desc = could not find container \"7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68\": container with ID starting with 7330ab3becc40e28ad1c142bfb8de093ff29bcf7a2f871ed534c69b7f742af68 not found: ID does not exist"
Oct 30 22:19:59.318784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318757    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.318908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318830    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.318908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318862    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.318908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318898    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318929    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.318973    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319001    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319045    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319057    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:59.319348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319163    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\") pod \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\" (UID: \"aaedf069-bcbb-4fb0-93ff-9fbe32a3b502\") "
Oct 30 22:19:59.319348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319176    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:59.319511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319480    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:59.319594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319495    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:59.319649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.319614    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:59.320099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.320075    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.320162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.320105    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.320162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.320121    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.320162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.320136    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.320162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.320150    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.321620 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.321593    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "kube-api-access-s8trj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:59.321766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.321618    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:59.321868 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.321852    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:59.322119 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.322104    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token" pod "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" (UID: "aaedf069-bcbb-4fb0-93ff-9fbe32a3b502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:59.422174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.422134    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.422174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.422169    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.422399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.422184    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.422399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.422198    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8trj\" (UniqueName: \"kubernetes.io/projected/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502-kube-api-access-s8trj\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.671755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.671723    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:19:59.677702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:19:59.677656    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67"]
Oct 30 22:20:01.494973 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:01.494943    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" path="/var/lib/kubelet/pods/aaedf069-bcbb-4fb0-93ff-9fbe32a3b502/volumes"
Oct 30 22:20:02.036579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.036547    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:20:02.131693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.130366    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.201736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.200211    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.201736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.200377    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.201736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.200147    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.221206    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.221777    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.221867    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.221906    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.222020    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.222085    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.229697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.222198    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\") pod \"3ad509fd-8dfd-47f9-847f-b07248210066\" (UID: \"3ad509fd-8dfd-47f9-847f-b07248210066\") "
Oct 30 22:20:02.239712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.236530    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:02.239712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.229105    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.239712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.236606    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.239712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.237183    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:02.239712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.237560    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:02.243002 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.242958    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:02.264140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.260418    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:02.276706 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.266206    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "kube-api-access-cnm2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:02.276706 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.266648    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token" pod "3ad509fd-8dfd-47f9-847f-b07248210066" (UID: "3ad509fd-8dfd-47f9-847f-b07248210066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:02.321017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.320976    4295 generic.go:356] "Generic (PLEG): container finished" podID="3ad509fd-8dfd-47f9-847f-b07248210066" containerID="40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786" exitCode=0
Oct 30 22:20:02.321219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.321071    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" event={"ID":"3ad509fd-8dfd-47f9-847f-b07248210066","Type":"ContainerDied","Data":"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"}
Oct 30 22:20:02.321219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.321122    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv" event={"ID":"3ad509fd-8dfd-47f9-847f-b07248210066","Type":"ContainerDied","Data":"536f6ac535830781988202d4ccd01b77a873850bd0e075c569ef266bc242ae98"}
Oct 30 22:20:02.321219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.321152    4295 scope.go:122] "RemoveContainer" containerID="40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"
Oct 30 22:20:02.321401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.321272    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337738    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnm2d\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-kube-api-access-cnm2d\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337764    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad509fd-8dfd-47f9-847f-b07248210066-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337781    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337795    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad509fd-8dfd-47f9-847f-b07248210066-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337809    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337821    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad509fd-8dfd-47f9-847f-b07248210066-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.337937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.337835    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad509fd-8dfd-47f9-847f-b07248210066-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:02.342646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.342626    4295 scope.go:122] "RemoveContainer" containerID="40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"
Oct 30 22:20:02.342941 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:20:02.342921    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786\": container with ID starting with 40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786 not found: ID does not exist" containerID="40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"
Oct 30 22:20:02.342990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.342948    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786"} err="failed to get container status \"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786\": rpc error: code = NotFound desc = could not find container \"40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786\": container with ID starting with 40a7166e93c31a038cbcb187b4cdc38e77af8d464338f1df36053602ff2db786 not found: ID does not exist"
Oct 30 22:20:02.525393 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.525253    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:20:02.537681 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:02.537288    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv"]
Oct 30 22:20:03.515690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:03.513565    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" path="/var/lib/kubelet/pods/3ad509fd-8dfd-47f9-847f-b07248210066/volumes"
Oct 30 22:20:08.246851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:08.246821    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.106143    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107228    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107251    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107314    4295 cpu_manager.go:414] "containerMap: removing container" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107323    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107509    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaedf069-bcbb-4fb0-93ff-9fbe32a3b502" containerName="istio-proxy"
Oct 30 22:20:11.110692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.107526    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad509fd-8dfd-47f9-847f-b07248210066" containerName="istio-proxy"
Oct 30 22:20:11.124726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.120972    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.208692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.208410    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4511b0\""
Oct 30 22:20:11.208692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.208603    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4511b0-dockercfg-t9lk2\""
Oct 30 22:20:11.274686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.274371    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.276851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.274933    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxt8j\" (UniqueName: \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.288906 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.288871    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:20:11.357594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.357555    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:20:11.364321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.364235    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:20:11.364479 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.364326    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.377097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.377066    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.377291 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.377274    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxt8j\" (UniqueName: \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.377996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.377973    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.440478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.440443    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxt8j\" (UniqueName: \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\") pod \"payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.473979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.473948    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:11.479529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.479496    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.479695 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.479674    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jn8s\" (UniqueName: \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.580727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.580693    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.580915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.580820    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jn8s\" (UniqueName: \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.583712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.581820    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.645175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.645097    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jn8s\" (UniqueName: \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\") pod \"payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.680526 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.680495    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:11.858834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:11.858804    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:20:11.859321 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:20:11.859298    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11aa727_f339_4663_a2b4_a4595ce8519a.slice/crio-50c1bb841fafd0d00e58a6486d1ca074a26c2cbefbf122a7fbe7f428b10086cc WatchSource:0}: Error finding container 50c1bb841fafd0d00e58a6486d1ca074a26c2cbefbf122a7fbe7f428b10086cc: Status 404 returned error can't find the container with id 50c1bb841fafd0d00e58a6486d1ca074a26c2cbefbf122a7fbe7f428b10086cc
Oct 30 22:20:12.063133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.063111    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:20:12.063520 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:20:12.063500    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc7251e_304c_4ae1_a2fc_f2e7132a1715.slice/crio-a9ae6fc1c7508cb4984510a96c0dd062022489b2b563b2ef9c234be934befc8e WatchSource:0}: Error finding container a9ae6fc1c7508cb4984510a96c0dd062022489b2b563b2ef9c234be934befc8e: Status 404 returned error can't find the container with id a9ae6fc1c7508cb4984510a96c0dd062022489b2b563b2ef9c234be934befc8e
Oct 30 22:20:12.405056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.405006    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" event={"ID":"b11aa727-f339-4663-a2b4-a4595ce8519a","Type":"ContainerStarted","Data":"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"}
Oct 30 22:20:12.405536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.405074    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:12.405536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.405099    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" event={"ID":"b11aa727-f339-4663-a2b4-a4595ce8519a","Type":"ContainerStarted","Data":"50c1bb841fafd0d00e58a6486d1ca074a26c2cbefbf122a7fbe7f428b10086cc"}
Oct 30 22:20:12.406848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.406818    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" event={"ID":"4bc7251e-304c-4ae1-a2fc-f2e7132a1715","Type":"ContainerStarted","Data":"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"}
Oct 30 22:20:12.406982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.406858    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" event={"ID":"4bc7251e-304c-4ae1-a2fc-f2e7132a1715","Type":"ContainerStarted","Data":"a9ae6fc1c7508cb4984510a96c0dd062022489b2b563b2ef9c234be934befc8e"}
Oct 30 22:20:12.406982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.406889    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:12.530544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.530494    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" podStartSLOduration=1.530478802 podStartE2EDuration="1.530478802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:12.528788485 +0000 UTC m=+1815.171382280" watchObservedRunningTime="2026-10-30 22:20:12.530478802 +0000 UTC m=+1815.173072587"
Oct 30 22:20:12.649905 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:12.649847    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" podStartSLOduration=1.649832119 podStartE2EDuration="1.649832119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:12.647772413 +0000 UTC m=+1815.290366203" watchObservedRunningTime="2026-10-30 22:20:12.649832119 +0000 UTC m=+1815.292425907"
Oct 30 22:20:23.208583 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.208547    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:20:23.212676 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.212645    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254460    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254603    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254694    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254746    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254922 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254904    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.254993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.254979    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.255081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.255048    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74gjn\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.255387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.255360    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.255487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.255461    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.268378 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.268354    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ff338b-openshift-default-dockercfg-cqdcm\""
Oct 30 22:20:23.323705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.321641    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.360699    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.360782    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.360875    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.360946    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.360971    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74gjn\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.361122    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.361151    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.361184    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.363691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.361257    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.414383 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.414202    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.414383 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.414251    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.422723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.422648    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:20:23.422914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.422866    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.423247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.423202    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.424148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.424121    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.424293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.424278    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:20:23.455652 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.455566    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.457698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.457649    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.466382 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.466329    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.494114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.494090    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74gjn\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\") pod \"e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") " pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.527566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.527539    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:23.627290 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:20:23.627255    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:20:23.980409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.980381    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:20:23.981984 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:20:23.981960    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9350b3_2120_461c_9eb3_4144ed6f0a2f.slice/crio-2ce5a4153c82899189b22b5116a48d8e03ccd5fa817e239f25afd0eb2761f460 WatchSource:0}: Error finding container 2ce5a4153c82899189b22b5116a48d8e03ccd5fa817e239f25afd0eb2761f460: Status 404 returned error can't find the container with id 2ce5a4153c82899189b22b5116a48d8e03ccd5fa817e239f25afd0eb2761f460
Oct 30 22:20:23.984902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.984866    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:23.984992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.984933    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:23.984992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:23.984962    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:24.487372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:24.487341    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerStarted","Data":"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"}
Oct 30 22:20:24.487864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:24.487394    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerStarted","Data":"2ce5a4153c82899189b22b5116a48d8e03ccd5fa817e239f25afd0eb2761f460"}
Oct 30 22:20:24.528530 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:24.528499    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:24.618719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:24.618650    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podStartSLOduration=1.6186362619999999 podStartE2EDuration="1.618636262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:24.615478456 +0000 UTC m=+1827.258072245" watchObservedRunningTime="2026-10-30 22:20:24.618636262 +0000 UTC m=+1827.261230050"
Oct 30 22:20:25.534431 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:25.534403    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:26.501486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:26.501455    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:26.502389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:26.502372    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:20:31.234796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.234756    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:20:31.239414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.239389    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.253174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.253149    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.253284 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.253188    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzf7\" (UniqueName: \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.432693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.398320    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ff338b-dockercfg-rxdvr\""
Oct 30 22:20:31.432693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.398464    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ff338b\""
Oct 30 22:20:31.432693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.398736    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.432693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.398802    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzf7\" (UniqueName: \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.432693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.400091    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.503723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.502228    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzf7\" (UniqueName: \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\") pod \"payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.569690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.564963    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:31.602943 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.602006    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:20:31.977703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:31.977656    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:20:31.978996 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:20:31.978971    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd5d603_0232_4fe8_a2d4_61857dd2de8f.slice/crio-2b194e532464fc59cf35e349108a0b31d0e50fc7f9e493203e243897b16c09f3 WatchSource:0}: Error finding container 2b194e532464fc59cf35e349108a0b31d0e50fc7f9e493203e243897b16c09f3: Status 404 returned error can't find the container with id 2b194e532464fc59cf35e349108a0b31d0e50fc7f9e493203e243897b16c09f3
Oct 30 22:20:32.615251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:32.615220    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" event={"ID":"3cd5d603-0232-4fe8-a2d4-61857dd2de8f","Type":"ContainerStarted","Data":"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"}
Oct 30 22:20:32.615707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:32.615269    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" event={"ID":"3cd5d603-0232-4fe8-a2d4-61857dd2de8f","Type":"ContainerStarted","Data":"2b194e532464fc59cf35e349108a0b31d0e50fc7f9e493203e243897b16c09f3"}
Oct 30 22:20:32.615707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:32.615368    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:32.738591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:32.738542    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" podStartSLOduration=1.7385296989999999 podStartE2EDuration="1.738529699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:32.736642074 +0000 UTC m=+1835.379235864" watchObservedRunningTime="2026-10-30 22:20:32.738529699 +0000 UTC m=+1835.381123487"
Oct 30 22:20:43.625858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:20:43.625782    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:20:53.610183 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:20:53.610143    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:21:01.315686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.280729    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:21:01.315686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.312929    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.386567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.385016    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:21:01.400298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.400270    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.400397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.400345    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lvh\" (UniqueName: \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.421451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.421425    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7cfd50-dockercfg-pcllc\""
Oct 30 22:21:01.421619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.421437    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7cfd50\""
Oct 30 22:21:01.501085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.501057    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.501243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.501128    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lvh\" (UniqueName: \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.501796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.501778    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.565814 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.565745    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lvh\" (UniqueName: \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\") pod \"payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:01.707612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:01.707582    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:02.088570 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:21:02.088541    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84a6549_e21a_453d_b229_5e8a3df1da20.slice/crio-a4421ca1e4f84469d2aec0e3a804da16f1fb6155eb80a10c9484c108e3ed0582 WatchSource:0}: Error finding container a4421ca1e4f84469d2aec0e3a804da16f1fb6155eb80a10c9484c108e3ed0582: Status 404 returned error can't find the container with id a4421ca1e4f84469d2aec0e3a804da16f1fb6155eb80a10c9484c108e3ed0582
Oct 30 22:21:02.089203 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:02.089179    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:21:02.874629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:02.874588    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" event={"ID":"f84a6549-e21a-453d-b229-5e8a3df1da20","Type":"ContainerStarted","Data":"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"}
Oct 30 22:21:02.875085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:02.874648    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" event={"ID":"f84a6549-e21a-453d-b229-5e8a3df1da20","Type":"ContainerStarted","Data":"a4421ca1e4f84469d2aec0e3a804da16f1fb6155eb80a10c9484c108e3ed0582"}
Oct 30 22:21:02.875085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:02.874701    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:03.020279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:03.000284    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" podStartSLOduration=2.000264656 podStartE2EDuration="2.000264656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:02.997052515 +0000 UTC m=+1865.639646306" watchObservedRunningTime="2026-10-30 22:21:03.000264656 +0000 UTC m=+1865.642858465"
Oct 30 22:21:13.884527 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:13.884495    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:21:21.585687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.585069    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:21:21.599271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.599242    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.611647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.611618    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:21:21.641790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.641755    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.641914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.641845    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.658493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.658460    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-18edc4\""
Oct 30 22:21:21.659591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.659520    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-18edc4-dockercfg-9t75h\""
Oct 30 22:21:21.737689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.736757    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:21:21.763554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.753174    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.763554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.753228    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.763554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.762882    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.766021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.765998    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:21.861108 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.860573    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:21.861108 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.860753    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-684s2\" (UniqueName: \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:21.864164 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.863034    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:21:21.918693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.918642    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\") pod \"payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:21.961793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.961757    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:21.961975 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.961873    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-684s2\" (UniqueName: \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:21.962417 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:21.962397    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:22.025534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:22.025497    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-684s2\" (UniqueName: \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\") pod \"payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") " pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:22.151155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:22.151085    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:22.228863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:22.226212    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:22.531433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:22.531409    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:21:22.532361 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:21:22.532337    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc657a8b_0d0d_4663_8bb7_b8bebe728549.slice/crio-46199962263db2417a7ab4c249fe93afc661ccd273dbd811a855a658aea69158 WatchSource:0}: Error finding container 46199962263db2417a7ab4c249fe93afc661ccd273dbd811a855a658aea69158: Status 404 returned error can't find the container with id 46199962263db2417a7ab4c249fe93afc661ccd273dbd811a855a658aea69158
Oct 30 22:21:22.632963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:22.632935    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:21:22.633307 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:21:22.633140    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ab2667_018e_4d64_a456_9c3c3d32899c.slice/crio-25963fe90b3adb9e72058debe67daeb8b052c5b36f29a9398b53369131f7f4e8 WatchSource:0}: Error finding container 25963fe90b3adb9e72058debe67daeb8b052c5b36f29a9398b53369131f7f4e8: Status 404 returned error can't find the container with id 25963fe90b3adb9e72058debe67daeb8b052c5b36f29a9398b53369131f7f4e8
Oct 30 22:21:23.076201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.076156    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" event={"ID":"65ab2667-018e-4d64-a456-9c3c3d32899c","Type":"ContainerStarted","Data":"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"}
Oct 30 22:21:23.076390 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.076222    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" event={"ID":"65ab2667-018e-4d64-a456-9c3c3d32899c","Type":"ContainerStarted","Data":"25963fe90b3adb9e72058debe67daeb8b052c5b36f29a9398b53369131f7f4e8"}
Oct 30 22:21:23.076390 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.076246    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:23.077938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.077914    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" event={"ID":"cc657a8b-0d0d-4663-8bb7-b8bebe728549","Type":"ContainerStarted","Data":"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"}
Oct 30 22:21:23.078046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.077952    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" event={"ID":"cc657a8b-0d0d-4663-8bb7-b8bebe728549","Type":"ContainerStarted","Data":"46199962263db2417a7ab4c249fe93afc661ccd273dbd811a855a658aea69158"}
Oct 30 22:21:23.078046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.077982    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:23.197488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.197434    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" podStartSLOduration=2.197420365 podStartE2EDuration="2.197420365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.195463668 +0000 UTC m=+1885.838057456" watchObservedRunningTime="2026-10-30 22:21:23.197420365 +0000 UTC m=+1885.840014154"
Oct 30 22:21:23.336737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:23.322476    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" podStartSLOduration=2.32245339 podStartE2EDuration="2.32245339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.317713954 +0000 UTC m=+1885.960307758" watchObservedRunningTime="2026-10-30 22:21:23.32245339 +0000 UTC m=+1885.965047182"
Oct 30 22:21:23.608249 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:21:23.608151    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:21:26.062762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.062727    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:21:26.067273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.067251    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.118707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.118649    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7kc2\" (UniqueName: \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.118840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.118714    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.118904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.118881    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.119031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.119011    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.174991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.174964    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-24aa4c-serving-cert\""
Oct 30 22:21:26.180023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.179204    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:21:26.220364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.220334    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7kc2\" (UniqueName: \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.220536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.220376    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.220536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.220463    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.220677 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.220555    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.220677 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:21:26.220629    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-serving-cert: secret "maas-api-e2e-shared-b-w6-24aa4c-serving-cert" not found
Oct 30 22:21:26.220790 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:21:26.220730    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls podName:4175c247-b10e-4ce3-8b86-4dbcb10ccffb nodeName:}" failed. No retries permitted until 2026-10-30 22:21:26.720708165 +0000 UTC m=+1889.363301951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls") pod "maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" (UID: "4175c247-b10e-4ce3-8b86-4dbcb10ccffb") : secret "maas-api-e2e-shared-b-w6-24aa4c-serving-cert" not found
Oct 30 22:21:26.221384 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.221356    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.223109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.223089    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.286855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.286828    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7kc2\" (UniqueName: \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.731578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.731529    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.734091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.734068    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:26.981573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:26.981543    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:27.366975 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:21:27.366949    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4175c247_b10e_4ce3_8b86_4dbcb10ccffb.slice/crio-aa6c01c5aeb0fffbe81fa191ad9549df5a5a9b59c3b6011ad14447476c801a92 WatchSource:0}: Error finding container aa6c01c5aeb0fffbe81fa191ad9549df5a5a9b59c3b6011ad14447476c801a92: Status 404 returned error can't find the container with id aa6c01c5aeb0fffbe81fa191ad9549df5a5a9b59c3b6011ad14447476c801a92
Oct 30 22:21:27.368713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:27.368684    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:21:28.117391 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:28.117352    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" event={"ID":"4175c247-b10e-4ce3-8b86-4dbcb10ccffb","Type":"ContainerStarted","Data":"aa6c01c5aeb0fffbe81fa191ad9549df5a5a9b59c3b6011ad14447476c801a92"}
Oct 30 22:21:29.125831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:29.125786    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" event={"ID":"4175c247-b10e-4ce3-8b86-4dbcb10ccffb","Type":"ContainerStarted","Data":"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"}
Oct 30 22:21:29.126307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:29.125889    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:29.248545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:29.248496    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" podStartSLOduration=2.230455039 podStartE2EDuration="3.248482244s" totalImagesPullingTime="1.018027205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:29.245755785 +0000 UTC m=+1891.888349575" watchObservedRunningTime="2026-10-30 22:21:29.248482244 +0000 UTC m=+1891.891076032"
Oct 30 22:21:34.089723 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:34.089691    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:21:34.090145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:34.089778    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:21:35.137519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:35.137486    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:21:53.168468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:53.168433    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:21:53.625384 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:21:53.625350    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:21:54.271055 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:54.271023    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:21:54.271451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:54.271234    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" containerID="cri-o://bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234" gracePeriod=30
Oct 30 22:21:54.568441 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:54.568357    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:21:54.568621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:54.568555    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" containerID="cri-o://7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715" gracePeriod=30
Oct 30 22:21:56.341633 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:56.341600    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:21:56.342141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:56.341652    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:58.169536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:58.169498    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing" containerID="cri-o://f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e" gracePeriod=25
Oct 30 22:21:58.250482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:58.250443    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:21:58.250641 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:58.250506    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:59.718154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.718127    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:21:59.786137 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786107    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786297 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786197    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786319    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786356    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786384    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786431    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786474    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786545    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.786791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.786773    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\") pod \"ff28f3af-74a4-42a5-997d-6efe9365edb3\" (UID: \"ff28f3af-74a4-42a5-997d-6efe9365edb3\") "
Oct 30 22:21:59.792021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.787526    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.792021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.791744    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.792298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.792273    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.795759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.795732    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.801030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.801007    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:59.820006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.819958    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:59.820858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.820819    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.821082 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.821049    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:59.822938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.822916    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp" pod "ff28f3af-74a4-42a5-997d-6efe9365edb3" (UID: "ff28f3af-74a4-42a5-997d-6efe9365edb3"). InnerVolumeSpecName "kube-api-access-mx9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:59.864465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.864441    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910490    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910524    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff28f3af-74a4-42a5-997d-6efe9365edb3-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910541    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910555    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910571    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910584    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff28f3af-74a4-42a5-997d-6efe9365edb3-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910598    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910612    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9cp\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-kube-api-access-mx9cp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.910713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:21:59.910628    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff28f3af-74a4-42a5-997d-6efe9365edb3-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.016152 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.016116    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.016453 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.016432    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.016700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.016624    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.017110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.017077    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.017306 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.017289    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.017440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.017425    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.017737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.017708    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.017935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.017918    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.018058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.018044    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\") pod \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\" (UID: \"d0f4016c-b847-4ee1-a28c-b819f8ce95af\") "
Oct 30 22:22:00.022325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.016746    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.022325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.019145    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.022325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.022022    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.022325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.022220    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:00.022961 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.022940    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.025022 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.025000    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:00.025913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.025886    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "kube-api-access-dsv5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:00.026820 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.026798    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:00.026993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.026967    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy" pod "d0f4016c-b847-4ee1-a28c-b819f8ce95af" (UID: "d0f4016c-b847-4ee1-a28c-b819f8ce95af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.120511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120423    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120461    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120476    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120492    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsv5t\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-kube-api-access-dsv5t\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120508    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120520    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120533    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f4016c-b847-4ee1-a28c-b819f8ce95af-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120549    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.120799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.120563    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f4016c-b847-4ee1-a28c-b819f8ce95af-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.352334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.352308    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"
Oct 30 22:22:00.352509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.352330    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" event={"ID":"d0f4016c-b847-4ee1-a28c-b819f8ce95af","Type":"ContainerDied","Data":"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"}
Oct 30 22:22:00.352509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.352390    4295 scope.go:122] "RemoveContainer" containerID="7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"
Oct 30 22:22:00.352509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.352264    4295 generic.go:356] "Generic (PLEG): container finished" podID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerID="7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715" exitCode=0
Oct 30 22:22:00.352509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.352492    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v" event={"ID":"d0f4016c-b847-4ee1-a28c-b819f8ce95af","Type":"ContainerDied","Data":"d576ee8b018b183887f7d265df6c5f18cd9cad348645a9d653bb4cb4063af0b8"}
Oct 30 22:22:00.354046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.354023    4295 generic.go:356] "Generic (PLEG): container finished" podID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerID="bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234" exitCode=0
Oct 30 22:22:00.354189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.354079    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"
Oct 30 22:22:00.354189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.354122    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" event={"ID":"ff28f3af-74a4-42a5-997d-6efe9365edb3","Type":"ContainerDied","Data":"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"}
Oct 30 22:22:00.354189 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.354158    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw" event={"ID":"ff28f3af-74a4-42a5-997d-6efe9365edb3","Type":"ContainerDied","Data":"f9d41773976eccf42d5db1b519cd21d04a2a7d2c06d8973eb6d5e0328ba0f57f"}
Oct 30 22:22:00.363155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.363134    4295 scope.go:122] "RemoveContainer" containerID="7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"
Oct 30 22:22:00.363500 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:00.363415    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715\": container with ID starting with 7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715 not found: ID does not exist" containerID="7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"
Oct 30 22:22:00.363573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.363452    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715"} err="failed to get container status \"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715\": rpc error: code = NotFound desc = could not find container \"7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715\": container with ID starting with 7ff00553707a79a4751eeed34b65ce4c6aeaa954db2db184d21f7e9debb17715 not found: ID does not exist"
Oct 30 22:22:00.363573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.363530    4295 scope.go:122] "RemoveContainer" containerID="bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"
Oct 30 22:22:00.373120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.373099    4295 scope.go:122] "RemoveContainer" containerID="bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"
Oct 30 22:22:00.373373 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:00.373353    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234\": container with ID starting with bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234 not found: ID does not exist" containerID="bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"
Oct 30 22:22:00.373434 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.373383    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234"} err="failed to get container status \"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234\": rpc error: code = NotFound desc = could not find container \"bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234\": container with ID starting with bcc43ffac33edce580df751db91b5357fb4582951c614568925cbd5658b35234 not found: ID does not exist"
Oct 30 22:22:00.531498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.531465    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:22:00.582026 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.581995    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v"]
Oct 30 22:22:00.665904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.665835    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:22:00.672050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:00.672022    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw"]
Oct 30 22:22:01.504269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:01.504233    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" path="/var/lib/kubelet/pods/d0f4016c-b847-4ee1-a28c-b819f8ce95af/volumes"
Oct 30 22:22:01.504739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:01.504724    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" path="/var/lib/kubelet/pods/ff28f3af-74a4-42a5-997d-6efe9365edb3/volumes"
Oct 30 22:22:02.544247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.544213    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:22:02.678828 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.678799    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:22:02.679385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679372    4295 cpu_manager.go:414] "containerMap: removing container" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy"
Oct 30 22:22:02.679427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679386    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy"
Oct 30 22:22:02.679427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679414    4295 cpu_manager.go:414] "containerMap: removing container" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy"
Oct 30 22:22:02.679427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679421    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy"
Oct 30 22:22:02.679532 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679523    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff28f3af-74a4-42a5-997d-6efe9365edb3" containerName="istio-proxy"
Oct 30 22:22:02.679568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.679537    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f4016c-b847-4ee1-a28c-b819f8ce95af" containerName="istio-proxy"
Oct 30 22:22:02.684346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.684322    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.742967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.742939    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:22:02.756759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.756725    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.756916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.756772    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v72d\" (UniqueName: \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.757056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.757030    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.757247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.757222    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.788443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.788419    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ff338b-serving-cert\""
Oct 30 22:22:02.858571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.858493    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.858752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.858603    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.858752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.858639    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.858752 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:02.858648    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-serving-cert: secret "maas-api-e2e-api-b-ff338b-serving-cert" not found
Oct 30 22:22:02.858752 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:02.858731    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls podName:c1020dfc-345f-42dd-953f-abae6b776f8d nodeName:}" failed. No retries permitted until 2026-10-30 22:22:03.358715675 +0000 UTC m=+1926.001309466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls") pod "maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" (UID: "c1020dfc-345f-42dd-953f-abae6b776f8d") : secret "maas-api-e2e-api-b-ff338b-serving-cert" not found
Oct 30 22:22:02.858995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.858676    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v72d\" (UniqueName: \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.859365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.859343    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.861174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.861155    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:02.948796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:02.948627    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v72d\" (UniqueName: \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:03.374483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:03.374446    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:03.377214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:03.377189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") pod \"maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:03.600118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:03.600089    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:04.069636 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:04.069602    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:22:04.074014 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:22:04.073983    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1020dfc_345f_42dd_953f_abae6b776f8d.slice/crio-3abf7e8f16328b0ad59c61d092487f0cef2cbb362e4e7d010e0b1daded81f14b WatchSource:0}: Error finding container 3abf7e8f16328b0ad59c61d092487f0cef2cbb362e4e7d010e0b1daded81f14b: Status 404 returned error can't find the container with id 3abf7e8f16328b0ad59c61d092487f0cef2cbb362e4e7d010e0b1daded81f14b
Oct 30 22:22:04.387084 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:04.387005    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" event={"ID":"c1020dfc-345f-42dd-953f-abae6b776f8d","Type":"ContainerStarted","Data":"3abf7e8f16328b0ad59c61d092487f0cef2cbb362e4e7d010e0b1daded81f14b"}
Oct 30 22:22:05.069137 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.069106    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:05.069572 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.069162    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:05.395049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.395018    4295 generic.go:356] "Generic (PLEG): container finished" podID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerID="2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb" exitCode=0
Oct 30 22:22:05.395216 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.395094    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerDied","Data":"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"}
Oct 30 22:22:05.395578 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.395560    4295 scope.go:122] "RemoveContainer" containerID="2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"
Oct 30 22:22:05.396805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.396783    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" event={"ID":"c1020dfc-345f-42dd-953f-abae6b776f8d","Type":"ContainerStarted","Data":"6b86d33e657e984d6fbb5ed98eb7c2cc978eac8bd9ef0af90ba7c81e61c7a9d0"}
Oct 30 22:22:05.396914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.396898    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:05.397445 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.397416    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:05.397510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.397492    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:05.397549 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.397534    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:05.695770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:05.695715    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" podStartSLOduration=2.882780881 podStartE2EDuration="3.69569918s" totalImagesPullingTime="812.918299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:05.693588679 +0000 UTC m=+1928.336182490" watchObservedRunningTime="2026-10-30 22:22:05.69569918 +0000 UTC m=+1928.338292971"
Oct 30 22:22:06.420962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:06.420921    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerStarted","Data":"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"}
Oct 30 22:22:06.524545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:06.524510    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:22:06.524847 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:06.524824    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:06.524908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:06.524876    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:07.525003 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:07.524974    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:07.525378 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:07.525035    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:08.525188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:08.525150    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:08.525598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:08.525203    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:09.524645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:09.524560    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:09.524645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:09.524618    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:10.524719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:10.524656    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:10.525100 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:10.524769    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:11.427502 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:11.427471    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:22:11.524871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:11.524844    4295 patch_prober.go:29] interesting pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:22:11.525286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:11.524889    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:22:12.529866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:12.529841    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:22:12.530340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:12.530319    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:22:12.530899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:12.530883    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:22:12.543636 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:12.543610    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:22:18.426919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.426862    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:22:18.548258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548220    4295 generic.go:356] "Generic (PLEG): container finished" podID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerID="f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e" exitCode=0
Oct 30 22:22:18.548454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548274    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"
Oct 30 22:22:18.548454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548282    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" event={"ID":"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd","Type":"ContainerDied","Data":"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"}
Oct 30 22:22:18.548454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548331    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8" event={"ID":"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd","Type":"ContainerDied","Data":"8076d34a5fa0e3645d041fe24d687a8d8897900a5de536209d0b6dcd45dd51bd"}
Oct 30 22:22:18.548454 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548358    4295 scope.go:122] "RemoveContainer" containerID="f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"
Oct 30 22:22:18.548863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.548843    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\" (UniqueName: \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\") pod \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") "
Oct 30 22:22:18.549178 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.549150    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\") pod \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\" (UID: \"4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd\") "
Oct 30 22:22:18.549719 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.549688    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume" pod "4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" (UID: "4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:18.551602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.551573    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd" pod "4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" (UID: "4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd"). InnerVolumeSpecName "kube-api-access-s4cmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:18.572870 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.572848    4295 scope.go:122] "RemoveContainer" containerID="f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"
Oct 30 22:22:18.573222 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:18.573194    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e\": container with ID starting with f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e not found: ID does not exist" containerID="f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"
Oct 30 22:22:18.573321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.573221    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e"} err="failed to get container status \"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e\": rpc error: code = NotFound desc = could not find container \"f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e\": container with ID starting with f6c9d05303d18e6b66fed8d66635331316898c3748398920d71d97285ccbbb3e not found: ID does not exist"
Oct 30 22:22:18.653158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.653109    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:18.653158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.653141    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4cmd\" (UniqueName: \"kubernetes.io/projected/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd-kube-api-access-s4cmd\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:18.974529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.974491    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:22:18.977805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:18.977777    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8"]
Oct 30 22:22:19.494003 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:19.493973    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" path="/var/lib/kubelet/pods/4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd/volumes"
Oct 30 22:22:23.608385 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:23.608348    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:22:33.515490 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.515461    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:22:33.516094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.516080    4295 cpu_manager.go:414] "containerMap: removing container" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing"
Oct 30 22:22:33.516141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.516095    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing"
Oct 30 22:22:33.516214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.516205    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="4613974b-3aad-4e2c-b8cb-fa3dd2b5efdd" containerName="payload-processing"
Oct 30 22:22:33.518647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.518626    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.550614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.550587    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.550746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.550652    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkph6\" (UniqueName: \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.550746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.550702    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.550836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.550818    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.575592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.575565    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7cfd50-serving-cert\""
Oct 30 22:22:33.628991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.628958    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:22:33.651902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.651865    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.652101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.652085    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.652219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.652200    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkph6\" (UniqueName: \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.652270 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.652251    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.652448 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:33.652435    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-serving-cert: secret "maas-api-e2e-shared-b-w3-7cfd50-serving-cert" not found
Oct 30 22:22:33.652519 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:33.652509    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls podName:bd4a76af-623b-460d-97f0-9e8bb6e4ecdc nodeName:}" failed. No retries permitted until 2026-10-30 22:22:34.152487524 +0000 UTC m=+1956.795081305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" (UID: "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc") : secret "maas-api-e2e-shared-b-w3-7cfd50-serving-cert" not found
Oct 30 22:22:33.653437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.653419    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.656439 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.656388    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:33.717227 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:33.717197    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkph6\" (UniqueName: \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:34.193388 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:34.193353    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:34.195858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:34.195833    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:34.431474 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:34.431441    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:34.813527 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:34.813497    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:22:34.814349 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:22:34.814324    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4a76af_623b_460d_97f0_9e8bb6e4ecdc.slice/crio-c4feddb152e762e6554ce997b717a35f4f27599b39b2bd899656bc45f0a0286e WatchSource:0}: Error finding container c4feddb152e762e6554ce997b717a35f4f27599b39b2bd899656bc45f0a0286e: Status 404 returned error can't find the container with id c4feddb152e762e6554ce997b717a35f4f27599b39b2bd899656bc45f0a0286e
Oct 30 22:22:35.675614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:35.675578    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" event={"ID":"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc","Type":"ContainerStarted","Data":"c4feddb152e762e6554ce997b717a35f4f27599b39b2bd899656bc45f0a0286e"}
Oct 30 22:22:36.683980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:36.683947    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" event={"ID":"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc","Type":"ContainerStarted","Data":"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"}
Oct 30 22:22:36.684380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:36.684101    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:36.884319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:36.884251    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" podStartSLOduration=2.846469545 podStartE2EDuration="3.88422935s" totalImagesPullingTime="1.037759805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:36.883001752 +0000 UTC m=+1959.525595542" watchObservedRunningTime="2026-10-30 22:22:36.88422935 +0000 UTC m=+1959.526823140"
Oct 30 22:22:42.717311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:42.717284    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:22:47.780445 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.780335    4295 generic.go:356] "Generic (PLEG): container finished" podID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerID="82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8" exitCode=0
Oct 30 22:22:47.780445 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.780414    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerDied","Data":"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"}
Oct 30 22:22:47.781044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.781002    4295 scope.go:122] "RemoveContainer" containerID="82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"
Oct 30 22:22:47.782964 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.782937    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:47.783056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.783008    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:47.783056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:47.783039    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:48.814403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:48.814353    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerStarted","Data":"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"}
Oct 30 22:22:49.540064 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:49.540033    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:22:49.540433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:49.540343    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:49.540614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:49.540589    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:50.528580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:50.528531    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:50.529112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:50.528594    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:51.528147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:51.528111    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:51.528332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:51.528178    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:52.528436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:52.528396    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:52.528894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:52.528465    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:53.532963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:53.532923    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:53.533611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:53.533576    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:53.635399 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:22:53.635363    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:22:54.532897 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:54.532860    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:22:54.533289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:54.533269    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:22:54.533953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:22:54.533937    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:23:01.692425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:01.692377    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:23:06.693049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.693011    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" podUID="bc82963e-4ebd-4965-8919-861c2bd880b9" containerName="payload-processing" containerID="cri-o://cc78964f664741e068af0b0dab526b50b2086c79740bd41874a2c9c9542abfe0" gracePeriod=25
Oct 30 22:23:06.947381 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.947314    4295 generic.go:356] "Generic (PLEG): container finished" podID="bc82963e-4ebd-4965-8919-861c2bd880b9" containerID="cc78964f664741e068af0b0dab526b50b2086c79740bd41874a2c9c9542abfe0" exitCode=0
Oct 30 22:23:06.947491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.947378    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" event={"ID":"bc82963e-4ebd-4965-8919-861c2bd880b9","Type":"ContainerDied","Data":"cc78964f664741e068af0b0dab526b50b2086c79740bd41874a2c9c9542abfe0"}
Oct 30 22:23:06.947491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.947430    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr" event={"ID":"bc82963e-4ebd-4965-8919-861c2bd880b9","Type":"ContainerDied","Data":"f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2"}
Oct 30 22:23:06.947491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.947452    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3a0fc7f366c67cabfc5b49b27e6c52549ce5a334ffa5d539075429e2b8a2bd2"
Oct 30 22:23:06.961519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:06.961493    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:23:07.101581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.101552    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\") pod \"bc82963e-4ebd-4965-8919-861c2bd880b9\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") "
Oct 30 22:23:07.101790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.101737    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\" (UniqueName: \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\") pod \"bc82963e-4ebd-4965-8919-861c2bd880b9\" (UID: \"bc82963e-4ebd-4965-8919-861c2bd880b9\") "
Oct 30 22:23:07.102015 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.101981    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume" pod "bc82963e-4ebd-4965-8919-861c2bd880b9" (UID: "bc82963e-4ebd-4965-8919-861c2bd880b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:07.102394 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.102369    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc82963e-4ebd-4965-8919-861c2bd880b9-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:07.104110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.104089    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx" pod "bc82963e-4ebd-4965-8919-861c2bd880b9" (UID: "bc82963e-4ebd-4965-8919-861c2bd880b9"). InnerVolumeSpecName "kube-api-access-xlfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:07.204275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.204195    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlfvx\" (UniqueName: \"kubernetes.io/projected/bc82963e-4ebd-4965-8919-861c2bd880b9-kube-api-access-xlfvx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:07.954170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:07.954139    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"
Oct 30 22:23:08.125594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:08.125561    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:23:08.175882 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:08.175852    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr"]
Oct 30 22:23:09.495059 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:23:09.495014    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc82963e-4ebd-4965-8919-861c2bd880b9" path="/var/lib/kubelet/pods/bc82963e-4ebd-4965-8919-861c2bd880b9/volumes"
Oct 30 22:23:23.609719 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:23:23.609651    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:23:53.665626 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:23:53.665535    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:24:10.084707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.084652    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:24:10.085124 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.084863    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" podUID="121a83ce-3115-40e6-8e13-984f099c6204" containerName="maas-api" containerID="cri-o://d2b23d56e6ea95d8ca2baf07ee0bc57db5ad3f806dfe4531f0c37a4889f27907" gracePeriod=30
Oct 30 22:24:10.434305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.433628    4295 generic.go:356] "Generic (PLEG): container finished" podID="121a83ce-3115-40e6-8e13-984f099c6204" containerID="d2b23d56e6ea95d8ca2baf07ee0bc57db5ad3f806dfe4531f0c37a4889f27907" exitCode=0
Oct 30 22:24:10.434305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.433703    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" event={"ID":"121a83ce-3115-40e6-8e13-984f099c6204","Type":"ContainerDied","Data":"d2b23d56e6ea95d8ca2baf07ee0bc57db5ad3f806dfe4531f0c37a4889f27907"}
Oct 30 22:24:10.434305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.433755    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv" event={"ID":"121a83ce-3115-40e6-8e13-984f099c6204","Type":"ContainerDied","Data":"28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d"}
Oct 30 22:24:10.434305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.433776    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28528c7175bbd9715d7da73365fc46ad6b9fc2f67964384d94c6d6eabd6c021d"
Oct 30 22:24:10.448153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.448127    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:24:10.606492 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.606452    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\" (UniqueName: \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\") pod \"121a83ce-3115-40e6-8e13-984f099c6204\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") "
Oct 30 22:24:10.606645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.606589    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\") pod \"121a83ce-3115-40e6-8e13-984f099c6204\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") "
Oct 30 22:24:10.606811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.606796    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\") pod \"121a83ce-3115-40e6-8e13-984f099c6204\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") "
Oct 30 22:24:10.606854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.606828    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\") pod \"121a83ce-3115-40e6-8e13-984f099c6204\" (UID: \"121a83ce-3115-40e6-8e13-984f099c6204\") "
Oct 30 22:24:10.608361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.608116    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca" pod "121a83ce-3115-40e6-8e13-984f099c6204" (UID: "121a83ce-3115-40e6-8e13-984f099c6204"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:10.610462 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.610434    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs" pod "121a83ce-3115-40e6-8e13-984f099c6204" (UID: "121a83ce-3115-40e6-8e13-984f099c6204"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:10.610851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.610814    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls" pod "121a83ce-3115-40e6-8e13-984f099c6204" (UID: "121a83ce-3115-40e6-8e13-984f099c6204"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:10.611007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.610982    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt" pod "121a83ce-3115-40e6-8e13-984f099c6204" (UID: "121a83ce-3115-40e6-8e13-984f099c6204"). InnerVolumeSpecName "kube-api-access-dspxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:10.733689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.730738    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.733689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.730772    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/121a83ce-3115-40e6-8e13-984f099c6204-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.733689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.730787    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a83ce-3115-40e6-8e13-984f099c6204-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.733689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:10.730800    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dspxt\" (UniqueName: \"kubernetes.io/projected/121a83ce-3115-40e6-8e13-984f099c6204-kube-api-access-dspxt\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:11.439876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:11.439850    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"
Oct 30 22:24:11.614988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:11.614948    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:24:11.665042 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:11.665007    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9b614c-7885668bfb-5z5wv"]
Oct 30 22:24:13.494685 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:13.494638    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121a83ce-3115-40e6-8e13-984f099c6204" path="/var/lib/kubelet/pods/121a83ce-3115-40e6-8e13-984f099c6204/volumes"
Oct 30 22:24:23.610970 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:23.610925    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:24:27.559945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:27.559910    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:24:27.678495 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:27.678464    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:24:32.560867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.560813    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" podUID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" containerName="payload-processing" containerID="cri-o://e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47" gracePeriod=25
Oct 30 22:24:32.679570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.679537    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" podUID="65ab2667-018e-4d64-a456-9c3c3d32899c" containerName="payload-pre-processing" containerID="cri-o://0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c" gracePeriod=25
Oct 30 22:24:32.836697 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.836641    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:24:32.911007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.910968    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\" (UniqueName: \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\") pod \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") "
Oct 30 22:24:32.911211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.911037    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\") pod \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\" (UID: \"cc657a8b-0d0d-4663-8bb7-b8bebe728549\") "
Oct 30 22:24:32.912094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.912064    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume" pod "cc657a8b-0d0d-4663-8bb7-b8bebe728549" (UID: "cc657a8b-0d0d-4663-8bb7-b8bebe728549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:32.919090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.919054    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2" pod "cc657a8b-0d0d-4663-8bb7-b8bebe728549" (UID: "cc657a8b-0d0d-4663-8bb7-b8bebe728549"). InnerVolumeSpecName "kube-api-access-684s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:32.955231 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:32.955208    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:24:33.012138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012102    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\") pod \"65ab2667-018e-4d64-a456-9c3c3d32899c\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") "
Oct 30 22:24:33.012322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012280    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\") pod \"65ab2667-018e-4d64-a456-9c3c3d32899c\" (UID: \"65ab2667-018e-4d64-a456-9c3c3d32899c\") "
Oct 30 22:24:33.012774 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012746    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume" pod "65ab2667-018e-4d64-a456-9c3c3d32899c" (UID: "65ab2667-018e-4d64-a456-9c3c3d32899c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:33.012941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012894    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ab2667-018e-4d64-a456-9c3c3d32899c-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:33.012941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012920    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-684s2\" (UniqueName: \"kubernetes.io/projected/cc657a8b-0d0d-4663-8bb7-b8bebe728549-kube-api-access-684s2\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:33.012941 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.012938    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657a8b-0d0d-4663-8bb7-b8bebe728549-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:33.014519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.014495    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz" pod "65ab2667-018e-4d64-a456-9c3c3d32899c" (UID: "65ab2667-018e-4d64-a456-9c3c3d32899c"). InnerVolumeSpecName "kube-api-access-zwqpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:33.114070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.113976    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:24:33.114580 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.114552    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/65ab2667-018e-4d64-a456-9c3c3d32899c-kube-api-access-zwqpz\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:33.660262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.660226    4295 generic.go:356] "Generic (PLEG): container finished" podID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" containerID="e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47" exitCode=0
Oct 30 22:24:33.660752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.660333    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"
Oct 30 22:24:33.660752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.660360    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" event={"ID":"cc657a8b-0d0d-4663-8bb7-b8bebe728549","Type":"ContainerDied","Data":"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"}
Oct 30 22:24:33.660752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.660427    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc" event={"ID":"cc657a8b-0d0d-4663-8bb7-b8bebe728549","Type":"ContainerDied","Data":"46199962263db2417a7ab4c249fe93afc661ccd273dbd811a855a658aea69158"}
Oct 30 22:24:33.660752 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.660452    4295 scope.go:122] "RemoveContainer" containerID="e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"
Oct 30 22:24:33.663198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.663069    4295 generic.go:356] "Generic (PLEG): container finished" podID="65ab2667-018e-4d64-a456-9c3c3d32899c" containerID="0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c" exitCode=0
Oct 30 22:24:33.663415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.663393    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" event={"ID":"65ab2667-018e-4d64-a456-9c3c3d32899c","Type":"ContainerDied","Data":"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"}
Oct 30 22:24:33.663598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.663578    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52" event={"ID":"65ab2667-018e-4d64-a456-9c3c3d32899c","Type":"ContainerDied","Data":"25963fe90b3adb9e72058debe67daeb8b052c5b36f29a9398b53369131f7f4e8"}
Oct 30 22:24:33.663737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.663719    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"
Oct 30 22:24:33.678497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.678472    4295 scope.go:122] "RemoveContainer" containerID="e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"
Oct 30 22:24:33.678937 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:33.678909    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47\": container with ID starting with e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47 not found: ID does not exist" containerID="e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"
Oct 30 22:24:33.679063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.678952    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47"} err="failed to get container status \"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47\": rpc error: code = NotFound desc = could not find container \"e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47\": container with ID starting with e1132293c6a68bf8ea52487e8348e80d24d9cf69e24ecf8f28c8a6249a89bd47 not found: ID does not exist"
Oct 30 22:24:33.679063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.678978    4295 scope.go:122] "RemoveContainer" containerID="0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"
Oct 30 22:24:33.696625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.696600    4295 scope.go:122] "RemoveContainer" containerID="0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"
Oct 30 22:24:33.697074 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:33.697034    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c\": container with ID starting with 0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c not found: ID does not exist" containerID="0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"
Oct 30 22:24:33.697170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.697077    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c"} err="failed to get container status \"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c\": rpc error: code = NotFound desc = could not find container \"0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c\": container with ID starting with 0ea6c15e9e67fa01b8c9cdecbba7a7b93c90e53fe81e55ef29ba8de55748fa1c not found: ID does not exist"
Oct 30 22:24:33.833101 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.833020    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:24:33.842483 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.842455    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc"]
Oct 30 22:24:33.959949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.959917    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:24:33.964613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:33.964588    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52"]
Oct 30 22:24:35.494809 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:35.494775    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ab2667-018e-4d64-a456-9c3c3d32899c" path="/var/lib/kubelet/pods/65ab2667-018e-4d64-a456-9c3c3d32899c/volumes"
Oct 30 22:24:35.495249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:35.495234    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" path="/var/lib/kubelet/pods/cc657a8b-0d0d-4663-8bb7-b8bebe728549/volumes"
Oct 30 22:24:38.115078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.115003    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" podUID="f84a6549-e21a-453d-b229-5e8a3df1da20" containerName="payload-processing" containerID="cri-o://f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c" gracePeriod=25
Oct 30 22:24:38.403759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.403734    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:24:38.516117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.516077    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\") pod \"f84a6549-e21a-453d-b229-5e8a3df1da20\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") "
Oct 30 22:24:38.516315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.516198    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\" (UniqueName: \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\") pod \"f84a6549-e21a-453d-b229-5e8a3df1da20\" (UID: \"f84a6549-e21a-453d-b229-5e8a3df1da20\") "
Oct 30 22:24:38.516475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.516451    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume" pod "f84a6549-e21a-453d-b229-5e8a3df1da20" (UID: "f84a6549-e21a-453d-b229-5e8a3df1da20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:38.516658 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.516642    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f84a6549-e21a-453d-b229-5e8a3df1da20-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:38.518416 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.518395    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh" pod "f84a6549-e21a-453d-b229-5e8a3df1da20" (UID: "f84a6549-e21a-453d-b229-5e8a3df1da20"). InnerVolumeSpecName "kube-api-access-b8lvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:38.618504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.618469    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lvh\" (UniqueName: \"kubernetes.io/projected/f84a6549-e21a-453d-b229-5e8a3df1da20-kube-api-access-b8lvh\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:38.749404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.749366    4295 generic.go:356] "Generic (PLEG): container finished" podID="f84a6549-e21a-453d-b229-5e8a3df1da20" containerID="f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c" exitCode=0
Oct 30 22:24:38.749604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.749425    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"
Oct 30 22:24:38.749604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.749460    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" event={"ID":"f84a6549-e21a-453d-b229-5e8a3df1da20","Type":"ContainerDied","Data":"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"}
Oct 30 22:24:38.749604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.749533    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5" event={"ID":"f84a6549-e21a-453d-b229-5e8a3df1da20","Type":"ContainerDied","Data":"a4421ca1e4f84469d2aec0e3a804da16f1fb6155eb80a10c9484c108e3ed0582"}
Oct 30 22:24:38.749604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.749554    4295 scope.go:122] "RemoveContainer" containerID="f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"
Oct 30 22:24:38.760284 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.760254    4295 scope.go:122] "RemoveContainer" containerID="f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"
Oct 30 22:24:38.760599 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:38.760577    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c\": container with ID starting with f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c not found: ID does not exist" containerID="f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"
Oct 30 22:24:38.760654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.760608    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c"} err="failed to get container status \"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c\": rpc error: code = NotFound desc = could not find container \"f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c\": container with ID starting with f0aaa93cdb5751b4d36947f6b41c7d32e8b128346050d79aa3a0c2b09adce55c not found: ID does not exist"
Oct 30 22:24:38.921312 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.921278    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:24:38.971546 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:38.971511    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5"]
Oct 30 22:24:39.495296 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:39.495262    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84a6549-e21a-453d-b229-5e8a3df1da20" path="/var/lib/kubelet/pods/f84a6549-e21a-453d-b229-5e8a3df1da20/volumes"
Oct 30 22:24:51.884233 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:51.884197    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:24:51.884713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:51.884465    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy" containerID="cri-o://02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971" gracePeriod=30
Oct 30 22:24:53.609035 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:53.608995    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:24:57.166956 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.166894    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:24:57.242443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242409    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242475    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242499    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242544    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242634    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242721    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242757    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242794    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.242921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242837    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.242921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.242870    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\") pod \"663f8c63-7a5e-4212-936d-7560f1daa7cd\" (UID: \"663f8c63-7a5e-4212-936d-7560f1daa7cd\") "
Oct 30 22:24:57.243167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243069    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.243298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243272    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.243506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243378    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:57.243506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243438    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.243629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243519    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.243629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.243599    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.245134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.245058    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "kube-api-access-xhqxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:57.245229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.245093    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:57.245341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.245320    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:57.245399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.245342    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token" pod "663f8c63-7a5e-4212-936d-7560f1daa7cd" (UID: "663f8c63-7a5e-4212-936d-7560f1daa7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:57.345144 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345108    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqxx\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-kube-api-access-xhqxx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345144 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345145    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345161    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663f8c63-7a5e-4212-936d-7560f1daa7cd-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345174    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345187    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345200    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.345364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.345213    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663f8c63-7a5e-4212-936d-7560f1daa7cd-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.928997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.928963    4295 generic.go:356] "Generic (PLEG): container finished" podID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerID="02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971" exitCode=0
Oct 30 22:24:57.929177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.929018    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"
Oct 30 22:24:57.929177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.929036    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerDied","Data":"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"}
Oct 30 22:24:57.929177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.929090    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr" event={"ID":"663f8c63-7a5e-4212-936d-7560f1daa7cd","Type":"ContainerDied","Data":"443f5204968d262911be17cd267d132e08d076d2e63ee74ac9e14a14086762f9"}
Oct 30 22:24:57.929177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.929112    4295 scope.go:122] "RemoveContainer" containerID="02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"
Oct 30 22:24:57.943061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.943038    4295 scope.go:122] "RemoveContainer" containerID="2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"
Oct 30 22:24:57.952853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.952828    4295 scope.go:122] "RemoveContainer" containerID="02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"
Oct 30 22:24:57.953157 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:57.953137    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971\": container with ID starting with 02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971 not found: ID does not exist" containerID="02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"
Oct 30 22:24:57.953205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.953171    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971"} err="failed to get container status \"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971\": rpc error: code = NotFound desc = could not find container \"02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971\": container with ID starting with 02f37c90a14c55180b3216f11695afecd58ac444e6ea265368e85beae1157971 not found: ID does not exist"
Oct 30 22:24:57.953205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.953190    4295 scope.go:122] "RemoveContainer" containerID="2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"
Oct 30 22:24:57.953461 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:24:57.953443    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb\": container with ID starting with 2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb not found: ID does not exist" containerID="2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"
Oct 30 22:24:57.953510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:57.953467    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb"} err="failed to get container status \"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb\": rpc error: code = NotFound desc = could not find container \"2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb\": container with ID starting with 2fb95e4d50ef0c0efd4d0a85ae0fd4b4712471224ad05cf4e2e0b4acdc4127fb not found: ID does not exist"
Oct 30 22:24:58.197934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:58.197855    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:24:58.247433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:58.247394    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr"]
Oct 30 22:24:58.765923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:58.765887    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:24:59.494744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:24:59.494708    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" path="/var/lib/kubelet/pods/663f8c63-7a5e-4212-936d-7560f1daa7cd/volumes"
Oct 30 22:25:04.852952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:04.852920    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:25:07.297498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:07.297462    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:25:07.417694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:07.417637    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:25:09.853650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:09.853545    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing" containerID="cri-o://ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af" gracePeriod=25
Oct 30 22:25:09.892826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:09.892792    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:25:10.005792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:10.005755    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:10.010103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:10.010073    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:25:12.298397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.298357    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" podUID="b11aa727-f339-4663-a2b4-a4595ce8519a" containerName="payload-processing" containerID="cri-o://2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2" gracePeriod=25
Oct 30 22:25:12.419093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.419013    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" podUID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" containerName="payload-pre-processing" containerID="cri-o://d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0" gracePeriod=25
Oct 30 22:25:12.583935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.583749    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:25:12.658872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.658836    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\") pod \"b11aa727-f339-4663-a2b4-a4595ce8519a\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") "
Oct 30 22:25:12.659070 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.658996    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\" (UniqueName: \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\") pod \"b11aa727-f339-4663-a2b4-a4595ce8519a\" (UID: \"b11aa727-f339-4663-a2b4-a4595ce8519a\") "
Oct 30 22:25:12.659275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.659243    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume" pod "b11aa727-f339-4663-a2b4-a4595ce8519a" (UID: "b11aa727-f339-4663-a2b4-a4595ce8519a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:12.659716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.659690    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11aa727-f339-4663-a2b4-a4595ce8519a-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.663796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.663770    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j" pod "b11aa727-f339-4663-a2b4-a4595ce8519a" (UID: "b11aa727-f339-4663-a2b4-a4595ce8519a"). InnerVolumeSpecName "kube-api-access-vxt8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:12.694133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.694107    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:25:12.760891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.760859    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\") pod \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") "
Oct 30 22:25:12.761057 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.761039    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\" (UniqueName: \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\") pod \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\" (UID: \"4bc7251e-304c-4ae1-a2fc-f2e7132a1715\") "
Oct 30 22:25:12.761329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.761302    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume" pod "4bc7251e-304c-4ae1-a2fc-f2e7132a1715" (UID: "4bc7251e-304c-4ae1-a2fc-f2e7132a1715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:12.761861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.761753    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxt8j\" (UniqueName: \"kubernetes.io/projected/b11aa727-f339-4663-a2b4-a4595ce8519a-kube-api-access-vxt8j\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.761861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.761780    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.763448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.763424    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s" pod "4bc7251e-304c-4ae1-a2fc-f2e7132a1715" (UID: "4bc7251e-304c-4ae1-a2fc-f2e7132a1715"). InnerVolumeSpecName "kube-api-access-8jn8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:12.863647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:12.863562    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jn8s\" (UniqueName: \"kubernetes.io/projected/4bc7251e-304c-4ae1-a2fc-f2e7132a1715-kube-api-access-8jn8s\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.137243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.137153    4295 generic.go:356] "Generic (PLEG): container finished" podID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" containerID="d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0" exitCode=0
Oct 30 22:25:13.137243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.137205    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"
Oct 30 22:25:13.137470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.137240    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" event={"ID":"4bc7251e-304c-4ae1-a2fc-f2e7132a1715","Type":"ContainerDied","Data":"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"}
Oct 30 22:25:13.137470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.137293    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh" event={"ID":"4bc7251e-304c-4ae1-a2fc-f2e7132a1715","Type":"ContainerDied","Data":"a9ae6fc1c7508cb4984510a96c0dd062022489b2b563b2ef9c234be934befc8e"}
Oct 30 22:25:13.137470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.137333    4295 scope.go:122] "RemoveContainer" containerID="d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"
Oct 30 22:25:13.138625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.138605    4295 generic.go:356] "Generic (PLEG): container finished" podID="b11aa727-f339-4663-a2b4-a4595ce8519a" containerID="2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2" exitCode=0
Oct 30 22:25:13.138766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.138651    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"
Oct 30 22:25:13.138766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.138707    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" event={"ID":"b11aa727-f339-4663-a2b4-a4595ce8519a","Type":"ContainerDied","Data":"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"}
Oct 30 22:25:13.138766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.138758    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z" event={"ID":"b11aa727-f339-4663-a2b4-a4595ce8519a","Type":"ContainerDied","Data":"50c1bb841fafd0d00e58a6486d1ca074a26c2cbefbf122a7fbe7f428b10086cc"}
Oct 30 22:25:13.149387 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.149366    4295 scope.go:122] "RemoveContainer" containerID="d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"
Oct 30 22:25:13.149699 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:13.149644    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0\": container with ID starting with d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0 not found: ID does not exist" containerID="d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"
Oct 30 22:25:13.149766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.149696    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0"} err="failed to get container status \"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0\": rpc error: code = NotFound desc = could not find container \"d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0\": container with ID starting with d9bc5734ee0b82ce5d8b77d7050dfd7fedacea0ac332578799a97bdd7775e8b0 not found: ID does not exist"
Oct 30 22:25:13.149766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.149724    4295 scope.go:122] "RemoveContainer" containerID="2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"
Oct 30 22:25:13.159916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.159899    4295 scope.go:122] "RemoveContainer" containerID="2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"
Oct 30 22:25:13.160226 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:13.160206    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2\": container with ID starting with 2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2 not found: ID does not exist" containerID="2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"
Oct 30 22:25:13.160293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.160238    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2"} err="failed to get container status \"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2\": rpc error: code = NotFound desc = could not find container \"2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2\": container with ID starting with 2f757be4cb03ed465360aba4f33d9f82545f07d6144ca6cbfe59aa10c85473e2 not found: ID does not exist"
Oct 30 22:25:13.310635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.310593    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:25:13.361997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.361965    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh"]
Oct 30 22:25:13.436303 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.436275    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:25:13.457309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.457260    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z"]
Oct 30 22:25:13.493884 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.493851    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" path="/var/lib/kubelet/pods/4bc7251e-304c-4ae1-a2fc-f2e7132a1715/volumes"
Oct 30 22:25:13.494329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.494315    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11aa727-f339-4663-a2b4-a4595ce8519a" path="/var/lib/kubelet/pods/b11aa727-f339-4663-a2b4-a4595ce8519a/volumes"
Oct 30 22:25:13.802614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.802525    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" containerID="cri-o://5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa" gracePeriod=45
Oct 30 22:25:13.974962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:13.974918    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 22:25:14.079900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.079872    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:25:14.087222 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087198    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087305    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087411    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087448    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087479    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.087515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087484    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\" (UniqueName: \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087565    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\") pod \"f5a15cfb-0ca8-4515-bb19-04400008e10f\" (UID: \"f5a15cfb-0ca8-4515-bb19-04400008e10f\") "
Oct 30 22:25:14.087824 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087767    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.087948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.087827    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.088153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.088099    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.088153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.088122    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.088153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.088136    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.089830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.089803    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.089939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.089880    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "kube-api-access-s8l6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.089985 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.089940    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs" pod "f5a15cfb-0ca8-4515-bb19-04400008e10f" (UID: "f5a15cfb-0ca8-4515-bb19-04400008e10f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:14.148097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.148061    4295 generic.go:356] "Generic (PLEG): container finished" podID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerID="5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa" exitCode=0
Oct 30 22:25:14.148319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.148104    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"
Oct 30 22:25:14.148319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.148147    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" event={"ID":"f5a15cfb-0ca8-4515-bb19-04400008e10f","Type":"ContainerDied","Data":"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"}
Oct 30 22:25:14.148319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.148205    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp" event={"ID":"f5a15cfb-0ca8-4515-bb19-04400008e10f","Type":"ContainerDied","Data":"2c5139fb096ac146c6f9b69a5345529313172845beed32c4ac261d05c2257119"}
Oct 30 22:25:14.148319 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.148228    4295 scope.go:122] "RemoveContainer" containerID="5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"
Oct 30 22:25:14.158218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.158195    4295 scope.go:122] "RemoveContainer" containerID="5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"
Oct 30 22:25:14.158483 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:14.158464    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa\": container with ID starting with 5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa not found: ID does not exist" containerID="5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"
Oct 30 22:25:14.158550 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.158490    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa"} err="failed to get container status \"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa\": rpc error: code = NotFound desc = could not find container \"5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa\": container with ID starting with 5e9db982dcb4ccf04f44625ddf87aa718c49758b0274f58f46630d85a6868cfa not found: ID does not exist"
Oct 30 22:25:14.189154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.189120    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a15cfb-0ca8-4515-bb19-04400008e10f-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.189154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.189155    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8l6t\" (UniqueName: \"kubernetes.io/projected/f5a15cfb-0ca8-4515-bb19-04400008e10f-kube-api-access-s8l6t\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.189367 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.189165    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a15cfb-0ca8-4515-bb19-04400008e10f-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.327008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.326783    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:25:14.335834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.335806    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-w4-bce513-kserve-dcd445b56-4qrfp"]
Oct 30 22:25:14.893126 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:14.893084    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" podUID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" containerName="payload-processing" containerID="cri-o://8a23b996a5ae4bc287478771045ce9b496c0154dc8dd281f638ae9173b6ec5f8" gracePeriod=25
Oct 30 22:25:15.011359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.011317    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" podUID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" containerName="payload-pre-processing" containerID="cri-o://455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e" gracePeriod=25
Oct 30 22:25:15.163646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.163564    4295 generic.go:356] "Generic (PLEG): container finished" podID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" containerID="8a23b996a5ae4bc287478771045ce9b496c0154dc8dd281f638ae9173b6ec5f8" exitCode=0
Oct 30 22:25:15.163899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.163861    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" event={"ID":"0fbf3240-f276-4e1d-9b02-9a43a95f09bc","Type":"ContainerDied","Data":"8a23b996a5ae4bc287478771045ce9b496c0154dc8dd281f638ae9173b6ec5f8"}
Oct 30 22:25:15.194688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.194633    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:25:15.216575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.216538    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\") pod \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") "
Oct 30 22:25:15.216840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.216820    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\" (UniqueName: \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\") pod \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\" (UID: \"0fbf3240-f276-4e1d-9b02-9a43a95f09bc\") "
Oct 30 22:25:15.216932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.216890    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume" pod "0fbf3240-f276-4e1d-9b02-9a43a95f09bc" (UID: "0fbf3240-f276-4e1d-9b02-9a43a95f09bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:15.217261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.217242    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.218937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.218918    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7" pod "0fbf3240-f276-4e1d-9b02-9a43a95f09bc" (UID: "0fbf3240-f276-4e1d-9b02-9a43a95f09bc"). InnerVolumeSpecName "kube-api-access-5msz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:15.309449 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.309416    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:25:15.319795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.319753    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5msz7\" (UniqueName: \"kubernetes.io/projected/0fbf3240-f276-4e1d-9b02-9a43a95f09bc-kube-api-access-5msz7\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.421054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.420968    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\" (UniqueName: \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\") pod \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") "
Oct 30 22:25:15.421054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.421050    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\") pod \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\" (UID: \"aa188cd6-41c9-4b34-b7c0-8de7673c7136\") "
Oct 30 22:25:15.421566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.421460    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume" pod "aa188cd6-41c9-4b34-b7c0-8de7673c7136" (UID: "aa188cd6-41c9-4b34-b7c0-8de7673c7136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:15.421566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.421541    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa188cd6-41c9-4b34-b7c0-8de7673c7136-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.423384 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.423357    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb" pod "aa188cd6-41c9-4b34-b7c0-8de7673c7136" (UID: "aa188cd6-41c9-4b34-b7c0-8de7673c7136"). InnerVolumeSpecName "kube-api-access-vt4bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:15.495428 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.495391    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" path="/var/lib/kubelet/pods/f5a15cfb-0ca8-4515-bb19-04400008e10f/volumes"
Oct 30 22:25:15.522545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:15.522516    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4bb\" (UniqueName: \"kubernetes.io/projected/aa188cd6-41c9-4b34-b7c0-8de7673c7136-kube-api-access-vt4bb\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.172853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.172817    4295 generic.go:356] "Generic (PLEG): container finished" podID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" containerID="455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e" exitCode=0
Oct 30 22:25:16.173036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.172869    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"
Oct 30 22:25:16.173036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.172896    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" event={"ID":"aa188cd6-41c9-4b34-b7c0-8de7673c7136","Type":"ContainerDied","Data":"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"}
Oct 30 22:25:16.173036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.172956    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk" event={"ID":"aa188cd6-41c9-4b34-b7c0-8de7673c7136","Type":"ContainerDied","Data":"1e0c720f7c490dae89c2ce23a729bc40ec336a759f37402cd59112909e6c270e"}
Oct 30 22:25:16.173036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.172988    4295 scope.go:122] "RemoveContainer" containerID="455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"
Oct 30 22:25:16.174539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.174510    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b" event={"ID":"0fbf3240-f276-4e1d-9b02-9a43a95f09bc","Type":"ContainerDied","Data":"28a779cc1d40875bedf004bce04784df93f01f592cb2e386a4381358b08788e3"}
Oct 30 22:25:16.174644 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.174627    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"
Oct 30 22:25:16.182684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.182650    4295 scope.go:122] "RemoveContainer" containerID="455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"
Oct 30 22:25:16.182963 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:16.182946    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e\": container with ID starting with 455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e not found: ID does not exist" containerID="455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"
Oct 30 22:25:16.183009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.182970    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e"} err="failed to get container status \"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e\": rpc error: code = NotFound desc = could not find container \"455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e\": container with ID starting with 455b6b7aca3d19db3881c71c1e40fe18f1159f1c236955cf2080e310c8e0155e not found: ID does not exist"
Oct 30 22:25:16.183009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.182988    4295 scope.go:122] "RemoveContainer" containerID="8a23b996a5ae4bc287478771045ce9b496c0154dc8dd281f638ae9173b6ec5f8"
Oct 30 22:25:16.393566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.393529    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:25:16.402965 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.402936    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk"]
Oct 30 22:25:16.521518 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.521483    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:25:16.527060 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:16.527036    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b"]
Oct 30 22:25:17.494418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:17.494389    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" path="/var/lib/kubelet/pods/0fbf3240-f276-4e1d-9b02-9a43a95f09bc/volumes"
Oct 30 22:25:17.494805 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:17.494791    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" path="/var/lib/kubelet/pods/aa188cd6-41c9-4b34-b7c0-8de7673c7136/volumes"
Oct 30 22:25:20.005759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:20.005715    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:23.610142 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:23.610099    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:25:25.716655 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:25.716617    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:25:25.717106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:25.716879    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" containerID="cri-o://63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd" gracePeriod=30
Oct 30 22:25:28.153538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:28.153496    4295 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:28.154058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:28.153573    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:29.113053 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:29.113012    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:25:29.113384 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:29.113333    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" containerID="cri-o://de77d2039e96297941af6f9d2fc0ba22343be2e0702c831cd7e78e5c4e17055d" gracePeriod=30
Oct 30 22:25:30.004960 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:25:30.004920    4295 logging.go:55] [core] [Channel #3166 SubChannel #3167]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.40:50052", ServerName: "10.133.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.40:50052: connect: connection refused"
Oct 30 22:25:30.120643 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.120582    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:25:30.249100 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.249069    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\" (UniqueName: \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\") pod \"1d64be76-3b79-413a-869d-4d9021a06e87\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") "
Oct 30 22:25:30.249300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.249138    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\") pod \"1d64be76-3b79-413a-869d-4d9021a06e87\" (UID: \"1d64be76-3b79-413a-869d-4d9021a06e87\") "
Oct 30 22:25:30.249500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.249476    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume" pod "1d64be76-3b79-413a-869d-4d9021a06e87" (UID: "1d64be76-3b79-413a-869d-4d9021a06e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:30.249603 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.249591    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d64be76-3b79-413a-869d-4d9021a06e87-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.251365 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.251342    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb" pod "1d64be76-3b79-413a-869d-4d9021a06e87" (UID: "1d64be76-3b79-413a-869d-4d9021a06e87"). InnerVolumeSpecName "kube-api-access-c4lrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.266082 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.266052    4295 generic.go:356] "Generic (PLEG): container finished" podID="1d64be76-3b79-413a-869d-4d9021a06e87" containerID="ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af" exitCode=0
Oct 30 22:25:30.266246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.266102    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" event={"ID":"1d64be76-3b79-413a-869d-4d9021a06e87","Type":"ContainerDied","Data":"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"}
Oct 30 22:25:30.266246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.266111    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"
Oct 30 22:25:30.266246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.266155    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" event={"ID":"1d64be76-3b79-413a-869d-4d9021a06e87","Type":"ContainerDied","Data":"f9aa55c09c7bf1332083d7b87faec97d887640086abbb1ba3a646ad95df909c0"}
Oct 30 22:25:30.266246 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.266184    4295 scope.go:122] "RemoveContainer" containerID="ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"
Oct 30 22:25:30.276921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.276895    4295 scope.go:122] "RemoveContainer" containerID="ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"
Oct 30 22:25:30.277291 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:30.277260    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af\": container with ID starting with ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af not found: ID does not exist" containerID="ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"
Oct 30 22:25:30.277379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.277300    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af"} err="failed to get container status \"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af\": rpc error: code = NotFound desc = could not find container \"ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af\": container with ID starting with ebacafe128b27b3f634d1568294eede53680a49675fd400b17c92822dbf1c8af not found: ID does not exist"
Oct 30 22:25:30.350442 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.350405    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lrb\" (UniqueName: \"kubernetes.io/projected/1d64be76-3b79-413a-869d-4d9021a06e87-kube-api-access-c4lrb\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.441451 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.441418    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:25:30.457192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.457143    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw"]
Oct 30 22:25:30.992460 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:30.992397    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:25:31.005258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.005220    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.40:50052\" within 1s: context deadline exceeded"
Oct 30 22:25:31.160439 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160402    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.160692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160525    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.160692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160565    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.160692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160594    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.160692 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160628    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160768    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160890    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160895    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.160983    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161044    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161179    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\") pod \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\" (UID: \"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5\") "
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161212    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.161544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161517    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:31.161995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161745    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.161995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161768    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.162102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161988    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.162102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.162008    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.162102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.161870    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.164041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.164019    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:31.164435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.164409    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.164528 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.164432    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.164571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.164546    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj" pod "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" (UID: "9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5"). InnerVolumeSpecName "kube-api-access-269lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.263478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.263360    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269lj\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-kube-api-access-269lj\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.263478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.263395    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.263478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.263409    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.263478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.263423    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.263478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.263436    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.274502 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.274471    4295 generic.go:356] "Generic (PLEG): container finished" podID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerID="63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd" exitCode=0
Oct 30 22:25:31.274654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.274515    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" event={"ID":"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5","Type":"ContainerDied","Data":"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"}
Oct 30 22:25:31.274654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.274529    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"
Oct 30 22:25:31.274654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.274558    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg" event={"ID":"9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5","Type":"ContainerDied","Data":"6f634e3a0c0579e35308a5513b70bd3b793df184fdc1830b2db488440a7caf04"}
Oct 30 22:25:31.274654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.274577    4295 scope.go:122] "RemoveContainer" containerID="63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"
Oct 30 22:25:31.285566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.285544    4295 scope.go:122] "RemoveContainer" containerID="63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"
Oct 30 22:25:31.285878 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:31.285856    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd\": container with ID starting with 63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd not found: ID does not exist" containerID="63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"
Oct 30 22:25:31.285949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.285889    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd"} err="failed to get container status \"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd\": rpc error: code = NotFound desc = could not find container \"63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd\": container with ID starting with 63232acaf50784a3e542f0b6a8aab8099e6541bea83d314afa6cf3902ef034cd not found: ID does not exist"
Oct 30 22:25:31.465276 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.465241    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:25:31.475409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.475380    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg"]
Oct 30 22:25:31.494650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.494617    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" path="/var/lib/kubelet/pods/1d64be76-3b79-413a-869d-4d9021a06e87/volumes"
Oct 30 22:25:31.495130 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:31.495115    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" path="/var/lib/kubelet/pods/9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5/volumes"
Oct 30 22:25:33.421876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.421843    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:25:33.422324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.422081    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" podUID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" containerName="maas-api" containerID="cri-o://994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea" gracePeriod=30
Oct 30 22:25:33.756553 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.756526    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:25:33.814090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814055    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\") pod \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") "
Oct 30 22:25:33.814274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814196    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\") pod \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") "
Oct 30 22:25:33.814274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814228    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\" (UniqueName: \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\") pod \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") "
Oct 30 22:25:33.814414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814388    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") pod \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\" (UID: \"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc\") "
Oct 30 22:25:33.814505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814442    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca" pod "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" (UID: "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:33.814928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.814908    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:33.816630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.816608    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs" pod "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" (UID: "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:33.816747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.816635    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls" pod "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" (UID: "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:33.816747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.816647    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6" pod "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" (UID: "bd4a76af-623b-460d-97f0-9e8bb6e4ecdc"). InnerVolumeSpecName "kube-api-access-vkph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:33.912439 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.910820    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:33.912439 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.910887    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:33.917430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.916927    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:33.917430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.916964    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkph6\" (UniqueName: \"kubernetes.io/projected/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-kube-api-access-vkph6\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:33.917430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:33.916983    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.323967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.322259    4295 generic.go:356] "Generic (PLEG): container finished" podID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" containerID="994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea" exitCode=0
Oct 30 22:25:34.323967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.322530    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" event={"ID":"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc","Type":"ContainerDied","Data":"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"}
Oct 30 22:25:34.323967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.322589    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm" event={"ID":"bd4a76af-623b-460d-97f0-9e8bb6e4ecdc","Type":"ContainerDied","Data":"c4feddb152e762e6554ce997b717a35f4f27599b39b2bd899656bc45f0a0286e"}
Oct 30 22:25:34.323967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.322721    4295 scope.go:122] "RemoveContainer" containerID="994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"
Oct 30 22:25:34.323967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.323365    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"
Oct 30 22:25:34.335547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.333210    4295 generic.go:356] "Generic (PLEG): container finished" podID="c5275837-506f-463f-9ed4-a1c77ee76482" containerID="de77d2039e96297941af6f9d2fc0ba22343be2e0702c831cd7e78e5c4e17055d" exitCode=0
Oct 30 22:25:34.335547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.333276    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" event={"ID":"c5275837-506f-463f-9ed4-a1c77ee76482","Type":"ContainerDied","Data":"de77d2039e96297941af6f9d2fc0ba22343be2e0702c831cd7e78e5c4e17055d"}
Oct 30 22:25:34.343544 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.343519    4295 scope.go:122] "RemoveContainer" containerID="994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"
Oct 30 22:25:34.344078 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:34.344008    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea\": container with ID starting with 994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea not found: ID does not exist" containerID="994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"
Oct 30 22:25:34.344207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.344095    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea"} err="failed to get container status \"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea\": rpc error: code = NotFound desc = could not find container \"994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea\": container with ID starting with 994317b0b4a1731440ff4d8bdbf7098c20c1fb842318f85377f125b731c861ea not found: ID does not exist"
Oct 30 22:25:34.402412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.402389    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531377    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531448    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531581    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531676    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531709    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531748    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531809    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531852    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.532689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.531878    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\") pod \"c5275837-506f-463f-9ed4-a1c77ee76482\" (UID: \"c5275837-506f-463f-9ed4-a1c77ee76482\") "
Oct 30 22:25:34.547765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.534357    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:34.547765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.535406    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:34.547765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.535479    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:34.547765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.535525    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:34.547765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.540337    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:34.549932 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.549724    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:34.550418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.550399    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:34.552622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.552602    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "kube-api-access-78724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:34.560806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.560002    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo" pod "c5275837-506f-463f-9ed4-a1c77ee76482" (UID: "c5275837-506f-463f-9ed4-a1c77ee76482"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:34.633552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633516    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633553    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5275837-506f-463f-9ed4-a1c77ee76482-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633568    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633585    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78724\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-kube-api-access-78724\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633600    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633612    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633625    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5275837-506f-463f-9ed4-a1c77ee76482-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633639    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5275837-506f-463f-9ed4-a1c77ee76482-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.633804 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.633652    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5275837-506f-463f-9ed4-a1c77ee76482-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.683519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.683487    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:25:34.688617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:34.688591    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7cfd50-66d85d4db7-m6ptm"]
Oct 30 22:25:35.341960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.341930    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"
Oct 30 22:25:35.341960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.341943    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s" event={"ID":"c5275837-506f-463f-9ed4-a1c77ee76482","Type":"ContainerDied","Data":"92fd6991932b275b9ea8c15e0987d98f184de2b6e439153cc102ae4214d3bf68"}
Oct 30 22:25:35.342224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.342017    4295 scope.go:122] "RemoveContainer" containerID="de77d2039e96297941af6f9d2fc0ba22343be2e0702c831cd7e78e5c4e17055d"
Oct 30 22:25:35.493750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.493720    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" path="/var/lib/kubelet/pods/bd4a76af-623b-460d-97f0-9e8bb6e4ecdc/volumes"
Oct 30 22:25:35.524505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.524470    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:25:35.574538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:35.574506    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s"]
Oct 30 22:25:37.494233 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:37.494199    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" path="/var/lib/kubelet/pods/c5275837-506f-463f-9ed4-a1c77ee76482/volumes"
Oct 30 22:25:43.774211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:43.774170    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:25:43.774624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:43.774371    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy" containerID="cri-o://a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8" gracePeriod=30
Oct 30 22:25:49.035589 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.035557    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:25:49.140496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140407    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140496 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140489    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140538    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140576    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140673    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140705    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140772    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140829    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.140952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140885    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:49.140952 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140918    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\") pod \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\" (UID: \"fe8070ff-d9cf-44c4-a3c1-466f1e64e751\") "
Oct 30 22:25:49.141164 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.140984    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:49.141164 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141111    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:49.141259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141183    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:49.141615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141578    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:49.141816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141794    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.141896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141822    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.141896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141839    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.141896 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.141855    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.143088 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.143060    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:49.143088 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.143082    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:49.143218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.143068    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "kube-api-access-pvn9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:49.143218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.143119    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy" pod "fe8070ff-d9cf-44c4-a3c1-466f1e64e751" (UID: "fe8070ff-d9cf-44c4-a3c1-466f1e64e751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:49.243401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.243366    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.243401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.243398    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.243401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.243408    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.243621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.243417    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvn9d\" (UniqueName: \"kubernetes.io/projected/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-kube-api-access-pvn9d\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.243621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.243428    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8070ff-d9cf-44c4-a3c1-466f1e64e751-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:49.507135 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.507106    4295 generic.go:356] "Generic (PLEG): container finished" podID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerID="a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8" exitCode=0
Oct 30 22:25:49.507268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.507154    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"
Oct 30 22:25:49.507268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.507187    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerDied","Data":"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"}
Oct 30 22:25:49.507268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.507240    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv" event={"ID":"fe8070ff-d9cf-44c4-a3c1-466f1e64e751","Type":"ContainerDied","Data":"efe53e01f661c86334d608f2e2bd43fede2fba33958390a8dff0e83d493fcabb"}
Oct 30 22:25:49.507268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.507266    4295 scope.go:122] "RemoveContainer" containerID="a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"
Oct 30 22:25:49.516823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.516800    4295 scope.go:122] "RemoveContainer" containerID="fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"
Oct 30 22:25:49.526110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.526090    4295 scope.go:122] "RemoveContainer" containerID="a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"
Oct 30 22:25:49.526359 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:49.526340    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8\": container with ID starting with a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8 not found: ID does not exist" containerID="a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"
Oct 30 22:25:49.526399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.526369    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8"} err="failed to get container status \"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8\": rpc error: code = NotFound desc = could not find container \"a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8\": container with ID starting with a61bbf34c5f1d9569dfa11dba3e6ed6b1fa6f71d984617b95c172981534226b8 not found: ID does not exist"
Oct 30 22:25:49.526399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.526390    4295 scope.go:122] "RemoveContainer" containerID="fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"
Oct 30 22:25:49.526657 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:49.526639    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91\": container with ID starting with fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91 not found: ID does not exist" containerID="fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"
Oct 30 22:25:49.526657 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.526733    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91"} err="failed to get container status \"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91\": rpc error: code = NotFound desc = could not find container \"fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91\": container with ID starting with fbff32e3bd5e85422fbd41d95e9a15db094625f33569b1878fb6d253d78c6e91 not found: ID does not exist"
Oct 30 22:25:49.682505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.682467    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:25:49.732631 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:49.732591    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv"]
Oct 30 22:25:51.494617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:51.494569    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" path="/var/lib/kubelet/pods/fe8070ff-d9cf-44c4-a3c1-466f1e64e751/volumes"
Oct 30 22:25:52.048061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.048027    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:25:52.048292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.048269    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" podUID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" containerName="maas-api" containerID="cri-o://7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c" gracePeriod=30
Oct 30 22:25:52.362980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.329614    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:25:52.385268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.384184    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\" (UniqueName: \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\") pod \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") "
Oct 30 22:25:52.385268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.384316    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\") pod \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") "
Oct 30 22:25:52.385268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.384366    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\") pod \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") "
Oct 30 22:25:52.385268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.384432    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") pod \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\" (UID: \"4175c247-b10e-4ce3-8b86-4dbcb10ccffb\") "
Oct 30 22:25:52.394942 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.394902    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca" pod "4175c247-b10e-4ce3-8b86-4dbcb10ccffb" (UID: "4175c247-b10e-4ce3-8b86-4dbcb10ccffb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:52.418920 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.418863    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls" pod "4175c247-b10e-4ce3-8b86-4dbcb10ccffb" (UID: "4175c247-b10e-4ce3-8b86-4dbcb10ccffb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:52.433755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.429632    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs" pod "4175c247-b10e-4ce3-8b86-4dbcb10ccffb" (UID: "4175c247-b10e-4ce3-8b86-4dbcb10ccffb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:52.433755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.429692    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2" pod "4175c247-b10e-4ce3-8b86-4dbcb10ccffb" (UID: "4175c247-b10e-4ce3-8b86-4dbcb10ccffb"). InnerVolumeSpecName "kube-api-access-w7kc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:52.486913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.486886    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:52.486913 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.486911    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:52.487102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.486928    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7kc2\" (UniqueName: \"kubernetes.io/projected/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-kube-api-access-w7kc2\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:52.487102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.486938    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175c247-b10e-4ce3-8b86-4dbcb10ccffb-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:52.565513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.565477    4295 generic.go:356] "Generic (PLEG): container finished" podID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" containerID="7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c" exitCode=0
Oct 30 22:25:52.566023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.565577    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" event={"ID":"4175c247-b10e-4ce3-8b86-4dbcb10ccffb","Type":"ContainerDied","Data":"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"}
Oct 30 22:25:52.566023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.565615    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz" event={"ID":"4175c247-b10e-4ce3-8b86-4dbcb10ccffb","Type":"ContainerDied","Data":"aa6c01c5aeb0fffbe81fa191ad9549df5a5a9b59c3b6011ad14447476c801a92"}
Oct 30 22:25:52.566023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.565635    4295 scope.go:122] "RemoveContainer" containerID="7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"
Oct 30 22:25:52.566023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.565770    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"
Oct 30 22:25:52.583646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.582741    4295 scope.go:122] "RemoveContainer" containerID="7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"
Oct 30 22:25:52.583646 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:52.583214    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c\": container with ID starting with 7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c not found: ID does not exist" containerID="7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"
Oct 30 22:25:52.583646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.583247    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c"} err="failed to get container status \"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c\": rpc error: code = NotFound desc = could not find container \"7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c\": container with ID starting with 7a6bd69ef520cc79b52e6eb2b76abfe3daf92c657b54fdc7c08167d27c987e4c not found: ID does not exist"
Oct 30 22:25:52.739630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.739563    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:25:52.742153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:52.742120    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-24aa4c-5fb89fcf58-vl9pz"]
Oct 30 22:25:53.494187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:53.494153    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" path="/var/lib/kubelet/pods/4175c247-b10e-4ce3-8b86-4dbcb10ccffb/volumes"
Oct 30 22:25:53.610024 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:25:53.609976    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:25:57.065159 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:57.065122    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:25:58.089878 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:58.089840    4295 scope.go:122] "RemoveContainer" containerID="d2b23d56e6ea95d8ca2baf07ee0bc57db5ad3f806dfe4531f0c37a4889f27907"
Oct 30 22:25:58.099598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:25:58.099571    4295 scope.go:122] "RemoveContainer" containerID="cc78964f664741e068af0b0dab526b50b2086c79740bd41874a2c9c9542abfe0"
Oct 30 22:26:02.066464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.066387    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" podUID="44d75289-e598-4146-85f2-c64ae0b3c60f" containerName="payload-processing" containerID="cri-o://e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b" gracePeriod=25
Oct 30 22:26:02.336537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.336508    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:26:02.483102 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.483066    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\" (UniqueName: \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\") pod \"44d75289-e598-4146-85f2-c64ae0b3c60f\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") "
Oct 30 22:26:02.483298 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.483284    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\") pod \"44d75289-e598-4146-85f2-c64ae0b3c60f\" (UID: \"44d75289-e598-4146-85f2-c64ae0b3c60f\") "
Oct 30 22:26:02.484046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.483991    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume" pod "44d75289-e598-4146-85f2-c64ae0b3c60f" (UID: "44d75289-e598-4146-85f2-c64ae0b3c60f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:02.495811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.489817    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk" pod "44d75289-e598-4146-85f2-c64ae0b3c60f" (UID: "44d75289-e598-4146-85f2-c64ae0b3c60f"). InnerVolumeSpecName "kube-api-access-687sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:02.584503 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.584427    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d75289-e598-4146-85f2-c64ae0b3c60f-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.584503 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.584453    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687sk\" (UniqueName: \"kubernetes.io/projected/44d75289-e598-4146-85f2-c64ae0b3c60f-kube-api-access-687sk\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.656632 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.656594    4295 generic.go:356] "Generic (PLEG): container finished" podID="44d75289-e598-4146-85f2-c64ae0b3c60f" containerID="e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b" exitCode=0
Oct 30 22:26:02.656855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.656652    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"
Oct 30 22:26:02.656855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.656656    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" event={"ID":"44d75289-e598-4146-85f2-c64ae0b3c60f","Type":"ContainerDied","Data":"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"}
Oct 30 22:26:02.656855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.656800    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6" event={"ID":"44d75289-e598-4146-85f2-c64ae0b3c60f","Type":"ContainerDied","Data":"566a417e33b9910238494d872ed070632b0d1137eecd45fbd218545e93c8963f"}
Oct 30 22:26:02.656855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.656829    4295 scope.go:122] "RemoveContainer" containerID="e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"
Oct 30 22:26:02.668147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.668124    4295 scope.go:122] "RemoveContainer" containerID="e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"
Oct 30 22:26:02.668498 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:02.668474    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b\": container with ID starting with e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b not found: ID does not exist" containerID="e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"
Oct 30 22:26:02.668568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.668508    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b"} err="failed to get container status \"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b\": rpc error: code = NotFound desc = could not find container \"e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b\": container with ID starting with e07f7e612eba681771a09123a46b2c05c35cab0362c3a58da486069cef5dd27b not found: ID does not exist"
Oct 30 22:26:02.827919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.827885    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:26:02.830180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:02.830160    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6"]
Oct 30 22:26:03.495493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:03.495457    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d75289-e598-4146-85f2-c64ae0b3c60f" path="/var/lib/kubelet/pods/44d75289-e598-4146-85f2-c64ae0b3c60f/volumes"
Oct 30 22:26:06.028650 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.028616    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:26:06.029156 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.028902    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" podUID="fa0d043d-2dc4-487b-a080-7c21f9183b84" containerName="maas-api" containerID="cri-o://5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab" gracePeriod=30
Oct 30 22:26:06.348326 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.348300    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:26:06.456693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.453157    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") pod \"fa0d043d-2dc4-487b-a080-7c21f9183b84\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") "
Oct 30 22:26:06.456693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.453265    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\") pod \"fa0d043d-2dc4-487b-a080-7c21f9183b84\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") "
Oct 30 22:26:06.456693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.453412    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\" (UniqueName: \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\") pod \"fa0d043d-2dc4-487b-a080-7c21f9183b84\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") "
Oct 30 22:26:06.456693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.453496    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\") pod \"fa0d043d-2dc4-487b-a080-7c21f9183b84\" (UID: \"fa0d043d-2dc4-487b-a080-7c21f9183b84\") "
Oct 30 22:26:06.456693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.454983    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca" pod "fa0d043d-2dc4-487b-a080-7c21f9183b84" (UID: "fa0d043d-2dc4-487b-a080-7c21f9183b84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:06.480885 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.473336    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls" pod "fa0d043d-2dc4-487b-a080-7c21f9183b84" (UID: "fa0d043d-2dc4-487b-a080-7c21f9183b84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:06.492557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.486161    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8" pod "fa0d043d-2dc4-487b-a080-7c21f9183b84" (UID: "fa0d043d-2dc4-487b-a080-7c21f9183b84"). InnerVolumeSpecName "kube-api-access-c5pq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:06.492557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.486271    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs" pod "fa0d043d-2dc4-487b-a080-7c21f9183b84" (UID: "fa0d043d-2dc4-487b-a080-7c21f9183b84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:06.555315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.555277    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.555315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.555318    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0d043d-2dc4-487b-a080-7c21f9183b84-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.555511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.555336    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5pq8\" (UniqueName: \"kubernetes.io/projected/fa0d043d-2dc4-487b-a080-7c21f9183b84-kube-api-access-c5pq8\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.555511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.555354    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0d043d-2dc4-487b-a080-7c21f9183b84-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.728283 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.728237    4295 generic.go:356] "Generic (PLEG): container finished" podID="fa0d043d-2dc4-487b-a080-7c21f9183b84" containerID="5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab" exitCode=0
Oct 30 22:26:06.728482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.728295    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"
Oct 30 22:26:06.728482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.728318    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" event={"ID":"fa0d043d-2dc4-487b-a080-7c21f9183b84","Type":"ContainerDied","Data":"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"}
Oct 30 22:26:06.728482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.728384    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82" event={"ID":"fa0d043d-2dc4-487b-a080-7c21f9183b84","Type":"ContainerDied","Data":"ac686aafc7b4f4869e3d0168f88a312383a83344e2c5477d5e715f86a01a2b9b"}
Oct 30 22:26:06.728482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.728411    4295 scope.go:122] "RemoveContainer" containerID="5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"
Oct 30 22:26:06.743598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.743577    4295 scope.go:122] "RemoveContainer" containerID="5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"
Oct 30 22:26:06.743995 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:06.743970    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab\": container with ID starting with 5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab not found: ID does not exist" containerID="5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"
Oct 30 22:26:06.744071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.744003    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab"} err="failed to get container status \"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab\": rpc error: code = NotFound desc = could not find container \"5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab\": container with ID starting with 5e7370382da7f7e71474ecdec7d934a1795f2de08f9b734d53ed4ddf9987abab not found: ID does not exist"
Oct 30 22:26:06.921539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.921507    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:26:06.926739 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:06.926711    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-512caa-995fbbbc4-wqp82"]
Oct 30 22:26:07.538149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:07.538114    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0d043d-2dc4-487b-a080-7c21f9183b84" path="/var/lib/kubelet/pods/fa0d043d-2dc4-487b-a080-7c21f9183b84/volumes"
Oct 30 22:26:12.941086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:12.941047    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:26:12.941689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:12.941311    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" podUID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" containerName="maas-api" containerID="cri-o://a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1" gracePeriod=30
Oct 30 22:26:13.267061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.267031    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:26:13.384353 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.383163    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\") pod \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") "
Oct 30 22:26:13.384353 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.383252    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") pod \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") "
Oct 30 22:26:13.384353 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.383322    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\") pod \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") "
Oct 30 22:26:13.384353 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.383367    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\" (UniqueName: \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\") pod \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\" (UID: \"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea\") "
Oct 30 22:26:13.391683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.390730    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472" pod "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" (UID: "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea"). InnerVolumeSpecName "kube-api-access-ds472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:13.391683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.391282    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca" pod "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" (UID: "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:13.397687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.396812    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls" pod "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" (UID: "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:13.401095 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.401052    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs" pod "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" (UID: "1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:13.485341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.485309    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:13.485341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.485335    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:13.485341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.485345    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:13.485577 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.485354    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds472\" (UniqueName: \"kubernetes.io/projected/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea-kube-api-access-ds472\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:13.769760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.769643    4295 generic.go:356] "Generic (PLEG): container finished" podID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" containerID="a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1" exitCode=0
Oct 30 22:26:13.769760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.769738    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"
Oct 30 22:26:13.769997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.769726    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" event={"ID":"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea","Type":"ContainerDied","Data":"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"}
Oct 30 22:26:13.769997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.769868    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9" event={"ID":"1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea","Type":"ContainerDied","Data":"189bb5e9dcd1b1a659bce76853458944879567b91f21f52d419f929f45c9d4e7"}
Oct 30 22:26:13.769997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.769907    4295 scope.go:122] "RemoveContainer" containerID="a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"
Oct 30 22:26:13.779423 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.779405    4295 scope.go:122] "RemoveContainer" containerID="a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"
Oct 30 22:26:13.779720 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:13.779700    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1\": container with ID starting with a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1 not found: ID does not exist" containerID="a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"
Oct 30 22:26:13.779785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.779730    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1"} err="failed to get container status \"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1\": rpc error: code = NotFound desc = could not find container \"a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1\": container with ID starting with a521b2c42b6292c8c59cec0bc1973763ec6a9d3d9926dbd3c426e069df7071a1 not found: ID does not exist"
Oct 30 22:26:13.985688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.979055    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:26:13.996824 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:13.996787    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce513-7f99c4bd84-47gb9"]
Oct 30 22:26:15.494289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:15.494245    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" path="/var/lib/kubelet/pods/1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea/volumes"
Oct 30 22:26:23.320512 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:23.320476    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:26:23.321020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:23.320721    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy" containerID="cri-o://9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98" gracePeriod=30
Oct 30 22:26:23.608486 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:23.608403    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:26:28.586871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.586810    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:26:28.725895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.725861    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.725909    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.725933    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.725959    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726067    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726090    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726122    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726161    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:28.726305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726176    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726313    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:28.726525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726322    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\") pod \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\" (UID: \"c0b17a3d-f7c9-4d07-be77-75eb94f623fc\") "
Oct 30 22:26:28.726525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726341    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:28.726525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726471    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:28.726783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.726746    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:28.727136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.727046    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.727136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.727074    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.727136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.727093    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.727136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.727109    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.727136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.727126    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.728541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.728511    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "kube-api-access-6g5g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:28.728649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.728568    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:28.728649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.728608    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:28.728806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.728717    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy" pod "c0b17a3d-f7c9-4d07-be77-75eb94f623fc" (UID: "c0b17a3d-f7c9-4d07-be77-75eb94f623fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:28.828219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.828178    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.828219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.828213    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.828219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.828224    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.828532 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.828232    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5g2\" (UniqueName: \"kubernetes.io/projected/c0b17a3d-f7c9-4d07-be77-75eb94f623fc-kube-api-access-6g5g2\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:28.857011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.856919    4295 generic.go:356] "Generic (PLEG): container finished" podID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerID="9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98" exitCode=0
Oct 30 22:26:28.857011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.856980    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"
Oct 30 22:26:28.857236 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.857007    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" event={"ID":"c0b17a3d-f7c9-4d07-be77-75eb94f623fc","Type":"ContainerDied","Data":"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"}
Oct 30 22:26:28.857236 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.857075    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb" event={"ID":"c0b17a3d-f7c9-4d07-be77-75eb94f623fc","Type":"ContainerDied","Data":"47a7c79e938bafcc3d7a0d1ea3036fa750003f412bcd7771462a6d72bb3af3b0"}
Oct 30 22:26:28.857236 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.857103    4295 scope.go:122] "RemoveContainer" containerID="9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"
Oct 30 22:26:28.869469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.869427    4295 scope.go:122] "RemoveContainer" containerID="9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"
Oct 30 22:26:28.869773 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:28.869740    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98\": container with ID starting with 9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98 not found: ID does not exist" containerID="9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"
Oct 30 22:26:28.869861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:28.869786    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98"} err="failed to get container status \"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98\": rpc error: code = NotFound desc = could not find container \"9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98\": container with ID starting with 9fcbe24c440aedb1a6569e057866c00fbd1cb7bcbc39ea5e90b4985f60364e98 not found: ID does not exist"
Oct 30 22:26:29.030761 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:29.030725    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:26:29.037166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:29.037136    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb"]
Oct 30 22:26:29.434130 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:29.434093    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:26:29.493994 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:29.493960    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" path="/var/lib/kubelet/pods/c0b17a3d-f7c9-4d07-be77-75eb94f623fc/volumes"
Oct 30 22:26:34.435476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.435401    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" podUID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" containerName="payload-pre-processing" containerID="cri-o://80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702" gracePeriod=25
Oct 30 22:26:34.836088 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.836053    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:26:34.839032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.838081    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\") pod \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") "
Oct 30 22:26:34.839032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.838305    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\" (UniqueName: \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\") pod \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\" (UID: \"3cd5d603-0232-4fe8-a2d4-61857dd2de8f\") "
Oct 30 22:26:34.839032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.838881    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume" pod "3cd5d603-0232-4fe8-a2d4-61857dd2de8f" (UID: "3cd5d603-0232-4fe8-a2d4-61857dd2de8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:34.840916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.840876    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7" pod "3cd5d603-0232-4fe8-a2d4-61857dd2de8f" (UID: "3cd5d603-0232-4fe8-a2d4-61857dd2de8f"). InnerVolumeSpecName "kube-api-access-jdzf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:34.940155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.940116    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdzf7\" (UniqueName: \"kubernetes.io/projected/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-kube-api-access-jdzf7\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:34.940155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.940144    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd5d603-0232-4fe8-a2d4-61857dd2de8f-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:34.945950 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.945918    4295 generic.go:356] "Generic (PLEG): container finished" podID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" containerID="80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702" exitCode=0
Oct 30 22:26:34.946117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.945958    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" event={"ID":"3cd5d603-0232-4fe8-a2d4-61857dd2de8f","Type":"ContainerDied","Data":"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"}
Oct 30 22:26:34.946117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.946017    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt" event={"ID":"3cd5d603-0232-4fe8-a2d4-61857dd2de8f","Type":"ContainerDied","Data":"2b194e532464fc59cf35e349108a0b31d0e50fc7f9e493203e243897b16c09f3"}
Oct 30 22:26:34.946117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.946040    4295 scope.go:122] "RemoveContainer" containerID="80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"
Oct 30 22:26:34.946299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.946164    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"
Oct 30 22:26:34.960249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.960231    4295 scope.go:122] "RemoveContainer" containerID="80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"
Oct 30 22:26:34.960688 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:34.960653    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702\": container with ID starting with 80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702 not found: ID does not exist" containerID="80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"
Oct 30 22:26:34.960776 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:34.960711    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702"} err="failed to get container status \"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702\": rpc error: code = NotFound desc = could not find container \"80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702\": container with ID starting with 80fe58ab5b9fbc54c8276b756277dd800aa1ad91cc5e84873c489dbb904cb702 not found: ID does not exist"
Oct 30 22:26:35.127915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:35.127792    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:26:35.131487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:35.131461    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt"]
Oct 30 22:26:35.529696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:35.529213    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" path="/var/lib/kubelet/pods/3cd5d603-0232-4fe8-a2d4-61857dd2de8f/volumes"
Oct 30 22:26:40.804722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:40.804632    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:26:40.805157 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:40.804874    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" podUID="18390268-bd86-4414-953e-be17364ef40f" containerName="maas-api" containerID="cri-o://e9944a096344a17c1adea7c45e1eb8703edb6a7415e624b72da053641a6e123d" gracePeriod=30
Oct 30 22:26:40.987738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:40.987594    4295 generic.go:356] "Generic (PLEG): container finished" podID="18390268-bd86-4414-953e-be17364ef40f" containerID="e9944a096344a17c1adea7c45e1eb8703edb6a7415e624b72da053641a6e123d" exitCode=0
Oct 30 22:26:40.987905 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:40.987833    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" event={"ID":"18390268-bd86-4414-953e-be17364ef40f","Type":"ContainerDied","Data":"e9944a096344a17c1adea7c45e1eb8703edb6a7415e624b72da053641a6e123d"}
Oct 30 22:26:41.112462 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.112437    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:26:41.219686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.214037    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\") pod \"18390268-bd86-4414-953e-be17364ef40f\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") "
Oct 30 22:26:41.219686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.214302    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\" (UniqueName: \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\") pod \"18390268-bd86-4414-953e-be17364ef40f\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") "
Oct 30 22:26:41.219686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.214345    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\") pod \"18390268-bd86-4414-953e-be17364ef40f\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") "
Oct 30 22:26:41.219686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.214389    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") pod \"18390268-bd86-4414-953e-be17364ef40f\" (UID: \"18390268-bd86-4414-953e-be17364ef40f\") "
Oct 30 22:26:41.219686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.219084    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca" pod "18390268-bd86-4414-953e-be17364ef40f" (UID: "18390268-bd86-4414-953e-be17364ef40f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:41.244780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.242214    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs" pod "18390268-bd86-4414-953e-be17364ef40f" (UID: "18390268-bd86-4414-953e-be17364ef40f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:41.244780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.242329    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7" pod "18390268-bd86-4414-953e-be17364ef40f" (UID: "18390268-bd86-4414-953e-be17364ef40f"). InnerVolumeSpecName "kube-api-access-g47q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:41.268429 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.268390    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls" pod "18390268-bd86-4414-953e-be17364ef40f" (UID: "18390268-bd86-4414-953e-be17364ef40f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:41.315915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.315880    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47q7\" (UniqueName: \"kubernetes.io/projected/18390268-bd86-4414-953e-be17364ef40f-kube-api-access-g47q7\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:41.315915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.315914    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18390268-bd86-4414-953e-be17364ef40f-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:41.316145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.315925    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:41.316145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.315934    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18390268-bd86-4414-953e-be17364ef40f-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:41.857789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.854838    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:26:41.857789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.855108    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" containerID="cri-o://6ced5b311968ee967cc82423b51547789b005d15ec33696fb02912365d84d496" gracePeriod=30
Oct 30 22:26:41.994498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.994456    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj" event={"ID":"18390268-bd86-4414-953e-be17364ef40f","Type":"ContainerDied","Data":"459c7ee135ff31668f337a83ebcad3b5692e6301d6759fb585984c4652ba2b9f"}
Oct 30 22:26:41.994498 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.994493    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"
Oct 30 22:26:41.994784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:41.994525    4295 scope.go:122] "RemoveContainer" containerID="e9944a096344a17c1adea7c45e1eb8703edb6a7415e624b72da053641a6e123d"
Oct 30 22:26:42.082262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:42.082232    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:26:42.082435 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:42.082284    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:26:42.166630 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:42.166539    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:26:42.171967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:42.171940    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c3d917-6cb8dcb57b-7qjvj"]
Oct 30 22:26:43.493881 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:43.493849    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18390268-bd86-4414-953e-be17364ef40f" path="/var/lib/kubelet/pods/18390268-bd86-4414-953e-be17364ef40f/volumes"
Oct 30 22:26:47.026180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.026140    4295 generic.go:356] "Generic (PLEG): container finished" podID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerID="6ced5b311968ee967cc82423b51547789b005d15ec33696fb02912365d84d496" exitCode=0
Oct 30 22:26:47.026582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.026215    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" event={"ID":"e48d60e1-9571-4ea7-a2c9-4558e71203a8","Type":"ContainerDied","Data":"6ced5b311968ee967cc82423b51547789b005d15ec33696fb02912365d84d496"}
Oct 30 22:26:47.123103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.123078    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:26:47.212850 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.212816    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.212865    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.212889    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.212915    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.212988    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213028    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213055    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213096    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213151    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.213324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213211    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\") pod \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\" (UID: \"e48d60e1-9571-4ea7-a2c9-4558e71203a8\") "
Oct 30 22:26:47.213543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213323    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.213543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213402    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:47.213651 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213621    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.213771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213690    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.213999 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.213976    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.214125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.214002    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.214125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.214015    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.214125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.214028    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.214125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.214040    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.215363 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.215336    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "kube-api-access-zpdz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:47.215525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.215504    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:47.215801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.215777    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.215801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.215782    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo" pod "e48d60e1-9571-4ea7-a2c9-4558e71203a8" (UID: "e48d60e1-9571-4ea7-a2c9-4558e71203a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:47.314773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.314743    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpdz4\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-kube-api-access-zpdz4\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.314773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.314772    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.314968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.314782    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.314968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:47.314792    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48d60e1-9571-4ea7-a2c9-4558e71203a8-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:48.033537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:48.033503    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"
Oct 30 22:26:48.034021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:48.033534    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh" event={"ID":"e48d60e1-9571-4ea7-a2c9-4558e71203a8","Type":"ContainerDied","Data":"14638b7c5fcf336e75eee04a2e2d49a8cebdfda295b8cf576e709b2ed00997c1"}
Oct 30 22:26:48.034021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:48.033619    4295 scope.go:122] "RemoveContainer" containerID="6ced5b311968ee967cc82423b51547789b005d15ec33696fb02912365d84d496"
Oct 30 22:26:48.254812 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:48.254779    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:26:48.309346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:48.309243    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh"]
Oct 30 22:26:49.494254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:49.494217    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" path="/var/lib/kubelet/pods/e48d60e1-9571-4ea7-a2c9-4558e71203a8/volumes"
Oct 30 22:26:53.609526 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:26:53.609487    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:26:55.134557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.134524    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:26:55.135020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135005    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing"
Oct 30 22:26:55.135062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135020    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing"
Oct 30 22:26:55.135062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135034    4295 cpu_manager.go:414] "containerMap: removing container" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:55.135062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135039    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:55.135062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135047    4295 cpu_manager.go:414] "containerMap: removing container" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135055    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135067    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135072    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135081    4295 cpu_manager.go:414] "containerMap: removing container" podUID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135086    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135094    4295 cpu_manager.go:414] "containerMap: removing container" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135099    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135106    4295 cpu_manager.go:414] "containerMap: removing container" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135110    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135118    4295 cpu_manager.go:414] "containerMap: removing container" podUID="65ab2667-018e-4d64-a456-9c3c3d32899c" containerName="payload-pre-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135123    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ab2667-018e-4d64-a456-9c3c3d32899c" containerName="payload-pre-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135133    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135140    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135153    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" containerName="maas-api"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135158    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" containerName="maas-api"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135166    4295 cpu_manager.go:414] "containerMap: removing container" podUID="44d75289-e598-4146-85f2-c64ae0b3c60f" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135171    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d75289-e598-4146-85f2-c64ae0b3c60f" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135177    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" containerName="payload-pre-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135181    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" containerName="payload-pre-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135187    4295 cpu_manager.go:414] "containerMap: removing container" podUID="18390268-bd86-4414-953e-be17364ef40f" containerName="maas-api"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135191    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18390268-bd86-4414-953e-be17364ef40f" containerName="maas-api"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135198    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f84a6549-e21a-453d-b229-5e8a3df1da20" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135202    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84a6549-e21a-453d-b229-5e8a3df1da20" containerName="payload-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135209    4295 cpu_manager.go:414] "containerMap: removing container" podUID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" containerName="payload-pre-processing"
Oct 30 22:26:55.135213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135217    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135231    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bc82963e-4ebd-4965-8919-861c2bd880b9" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135235    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc82963e-4ebd-4965-8919-861c2bd880b9" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135242    4295 cpu_manager.go:414] "containerMap: removing container" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135246    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135255    4295 cpu_manager.go:414] "containerMap: removing container" podUID="fa0d043d-2dc4-487b-a080-7c21f9183b84" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135258    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0d043d-2dc4-487b-a080-7c21f9183b84" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135264    4295 cpu_manager.go:414] "containerMap: removing container" podUID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135268    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135276    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135280    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135286    4295 cpu_manager.go:414] "containerMap: removing container" podUID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135291    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135305    4295 cpu_manager.go:414] "containerMap: removing container" podUID="121a83ce-3115-40e6-8e13-984f099c6204" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135309    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121a83ce-3115-40e6-8e13-984f099c6204" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135315    4295 cpu_manager.go:414] "containerMap: removing container" podUID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135319    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135326    4295 cpu_manager.go:414] "containerMap: removing container" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135331    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135338    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b11aa727-f339-4663-a2b4-a4595ce8519a" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135342    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11aa727-f339-4663-a2b4-a4595ce8519a" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135348    4295 cpu_manager.go:414] "containerMap: removing container" podUID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135352    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135425    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b17a3d-f7c9-4d07-be77-75eb94f623fc" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135435    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbf3240-f276-4e1d-9b02-9a43a95f09bc" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135440    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="121a83ce-3115-40e6-8e13-984f099c6204" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135446    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="4175c247-b10e-4ce3-8b86-4dbcb10ccffb" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135452    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcf0aab-a38d-4e10-b8b4-a140f4ecaba5" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135463    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11aa727-f339-4663-a2b4-a4595ce8519a" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135471    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc7251e-304c-4ae1-a2fc-f2e7132a1715" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135479    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d64be76-3b79-413a-869d-4d9021a06e87" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135484    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135489    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135495    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5275837-506f-463f-9ed4-a1c77ee76482" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135500    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc657a8b-0d0d-4663-8bb7-b8bebe728549" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135506    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48d60e1-9571-4ea7-a2c9-4558e71203a8" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135511    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8070ff-d9cf-44c4-a3c1-466f1e64e751" containerName="istio-proxy"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135517    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ab2667-018e-4d64-a456-9c3c3d32899c" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135523    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a15cfb-0ca8-4515-bb19-04400008e10f" containerName="main"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135528    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea54f7f-4cd0-4d07-b822-ce9603f8a4ea" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135534    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d75289-e598-4146-85f2-c64ae0b3c60f" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135543    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd5d603-0232-4fe8-a2d4-61857dd2de8f" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135551    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="18390268-bd86-4414-953e-be17364ef40f" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135558    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84a6549-e21a-453d-b229-5e8a3df1da20" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135563    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa188cd6-41c9-4b34-b7c0-8de7673c7136" containerName="payload-pre-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135568    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc82963e-4ebd-4965-8919-861c2bd880b9" containerName="payload-processing"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135574    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4a76af-623b-460d-97f0-9e8bb6e4ecdc" containerName="maas-api"
Oct 30 22:26:55.135939 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.135579    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0d043d-2dc4-487b-a080-7c21f9183b84" containerName="maas-api"
Oct 30 22:26:55.138688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.138651    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242272    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242320    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242457    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242480    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242506    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242533    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsznt\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242703    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242783    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.242901 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.242883    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343562    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343647    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343703    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343753    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343781    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.343807 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343805    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343887    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsznt\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343945    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344117 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.343979    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344291 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.344189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.344373    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.344414    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.344432    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.344840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.344814    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.346332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.346308    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.346562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.346538    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.416405 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.416316    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:26:55.432554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.432512    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.432955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.432929    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsznt\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-99f4v\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.451983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.451952    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:55.884001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.883976    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:26:55.885384 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:26:55.885347    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989cbd04_3b28_461a_9448_a2736c8f7aa2.slice/crio-135197432af4f0068bc5e96c6489a624305c1868e7b434b61c61dbe9cbf02a51 WatchSource:0}: Error finding container 135197432af4f0068bc5e96c6489a624305c1868e7b434b61c61dbe9cbf02a51: Status 404 returned error can't find the container with id 135197432af4f0068bc5e96c6489a624305c1868e7b434b61c61dbe9cbf02a51
Oct 30 22:26:55.887734 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.887700    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:55.887798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.887768    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:55.887798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:55.887794    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:56.081747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.081710    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" event={"ID":"989cbd04-3b28-461a-9448-a2736c8f7aa2","Type":"ContainerStarted","Data":"c33aeb3df5db1973dbcc4140cba784ce4c2a5d6f9a38763c8f4a7dccef85de25"}
Oct 30 22:26:56.081892 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.081763    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" event={"ID":"989cbd04-3b28-461a-9448-a2736c8f7aa2","Type":"ContainerStarted","Data":"135197432af4f0068bc5e96c6489a624305c1868e7b434b61c61dbe9cbf02a51"}
Oct 30 22:26:56.208038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.207969    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podStartSLOduration=1.207947421 podStartE2EDuration="1.207947421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:56.206122693 +0000 UTC m=+2218.848716507" watchObservedRunningTime="2026-10-30 22:26:56.207947421 +0000 UTC m=+2218.850541213"
Oct 30 22:26:56.452290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.452258    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:26:56.453740 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.453708    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:26:56.453872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.453767    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:26:56.984190 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:56.984148    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:26:57.452497 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:57.452462    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:26:57.452921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:57.452519    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:26:58.452313 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:58.452287    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:26:58.452520 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:58.452337    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:26:59.452842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.452804    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:26:59.453233 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.452858    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:26:59.866336 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.866242    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"]
Oct 30 22:26:59.866912 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.866892    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="663f8c63-7a5e-4212-936d-7560f1daa7cd" containerName="istio-proxy"
Oct 30 22:26:59.871526 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.871507    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.919523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.919488    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"]
Oct 30 22:26:59.923191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.923165    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:26:59.923833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.923804    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vk7\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-kube-api-access-64vk7\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.923957 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.923935    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.923996 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.923980    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924085    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924180    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310492ba-b2e0-4610-90f6-6fb5d5deb753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924236    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924314    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924359    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.924523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.924505    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:26:59.978299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:26:59.978266    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"]
Oct 30 22:27:00.031452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031406    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310492ba-b2e0-4610-90f6-6fb5d5deb753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.031618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031487    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031525    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.031618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031553    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031579    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.031618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031608    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031649    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66k9t\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-kube-api-access-66k9t\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031769    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031806    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031833    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031862    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.031949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.031910    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vk7\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-kube-api-access-64vk7\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032009    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032039    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032124    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032154    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032192    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0403e8-735c-4807-92d8-7a0fed332e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.032255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.032226    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.033530 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.033497    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310492ba-b2e0-4610-90f6-6fb5d5deb753-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.034180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.034153    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.034376 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.034352    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.035290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.035264    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.036008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.035699    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.036008 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.035928    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310492ba-b2e0-4610-90f6-6fb5d5deb753-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.039018 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.038828    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.056731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.056703    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"]
Oct 30 22:27:00.117077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.116901    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.118832 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.118803    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vk7\" (UniqueName: \"kubernetes.io/projected/310492ba-b2e0-4610-90f6-6fb5d5deb753-kube-api-access-64vk7\") pod \"maas-default-gateway-openshift-default-67bf986f99-whxxj\" (UID: \"310492ba-b2e0-4610-90f6-6fb5d5deb753\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.133165 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133126    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133193    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133238    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66k9t\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-kube-api-access-66k9t\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133271    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133300    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133327    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133465    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133494    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0403e8-735c-4807-92d8-7a0fed332e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133530    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.133911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.133894    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.136299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.135691    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.136299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.135942    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.136299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.136217    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.136299 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.136257    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0403e8-735c-4807-92d8-7a0fed332e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.136843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.136819    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.137706 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.137683    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.184341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.184313    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:00.215278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.215234    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.215757 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.215729    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66k9t\" (UniqueName: \"kubernetes.io/projected/7a0403e8-735c-4807-92d8-7a0fed332e42-kube-api-access-66k9t\") pod \"maas-default-gateway-openshift-default-67bf986f99-5l25z\" (UID: \"7a0403e8-735c-4807-92d8-7a0fed332e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.236754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.236727    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:00.452563 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.452530    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:27:00.452746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.452581    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:27:00.641067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.641032    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"]
Oct 30 22:27:00.678427 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:27:00.678396    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310492ba_b2e0_4610_90f6_6fb5d5deb753.slice/crio-73ff4e55d50b6c99c404ffbb7601a989ee6342f62db2102d58a333a132ef1a47 WatchSource:0}: Error finding container 73ff4e55d50b6c99c404ffbb7601a989ee6342f62db2102d58a333a132ef1a47: Status 404 returned error can't find the container with id 73ff4e55d50b6c99c404ffbb7601a989ee6342f62db2102d58a333a132ef1a47
Oct 30 22:27:00.681646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.681610    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.681764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.681703    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.681764 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.681743    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.714556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.714467    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"]
Oct 30 22:27:00.718744 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:27:00.718716    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0403e8_735c_4807_92d8_7a0fed332e42.slice/crio-7baa5da6baba3c3a1da0911d52e799d2dc92707aade8fa390525fee366acecc8 WatchSource:0}: Error finding container 7baa5da6baba3c3a1da0911d52e799d2dc92707aade8fa390525fee366acecc8: Status 404 returned error can't find the container with id 7baa5da6baba3c3a1da0911d52e799d2dc92707aade8fa390525fee366acecc8
Oct 30 22:27:00.721217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.721180    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.721345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.721254    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.721345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:00.721284    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:01.144741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.144633    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" event={"ID":"7a0403e8-735c-4807-92d8-7a0fed332e42","Type":"ContainerStarted","Data":"0a3215d8cefdf15c2e18f26a36929973e19a80b2b7f06278da23bd02f5a3392b"}
Oct 30 22:27:01.145242 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.145200    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" event={"ID":"7a0403e8-735c-4807-92d8-7a0fed332e42","Type":"ContainerStarted","Data":"7baa5da6baba3c3a1da0911d52e799d2dc92707aade8fa390525fee366acecc8"}
Oct 30 22:27:01.146899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.146870    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" event={"ID":"310492ba-b2e0-4610-90f6-6fb5d5deb753","Type":"ContainerStarted","Data":"4cd943e0572530ef9b88e67b148833c4f88a63c9ca956da7ca2e65bc8db8d1e4"}
Oct 30 22:27:01.146998 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.146921    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" event={"ID":"310492ba-b2e0-4610-90f6-6fb5d5deb753","Type":"ContainerStarted","Data":"73ff4e55d50b6c99c404ffbb7601a989ee6342f62db2102d58a333a132ef1a47"}
Oct 30 22:27:01.184838 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.184806    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:01.186539 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.186509    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:01.186712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.186559    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:01.237043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.237005    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:01.239267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.239235    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:01.239411 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.239299    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:01.278750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.278687    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podStartSLOduration=2.2786525060000002 podStartE2EDuration="2.278652506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:01.27620703 +0000 UTC m=+2223.918800833" watchObservedRunningTime="2026-10-30 22:27:01.278652506 +0000 UTC m=+2223.921246296"
Oct 30 22:27:01.453109 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.453074    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:27:01.453290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:01.453128    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:27:02.185340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:02.185292    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:02.185876 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:02.185349    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:02.244373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:02.244186    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:02.244373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:02.244255    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:03.081914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.081878    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:27:03.082552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.082528    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:27:03.084842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.084821    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:27:03.185493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.185459    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:03.186001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.185520    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:03.237436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.237397    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:03.237762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.237732    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:03.333741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:03.310957    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podStartSLOduration=4.310935141 podStartE2EDuration="4.310935141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:01.409879729 +0000 UTC m=+2224.052473518" watchObservedRunningTime="2026-10-30 22:27:03.310935141 +0000 UTC m=+2225.953528933"
Oct 30 22:27:04.185068 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:04.185038    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:04.185230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:04.185088    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:04.238118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:04.238077    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:04.238749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:04.238715    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:05.185367 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:05.185335    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:05.185639 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:05.185617    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:05.240794 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:05.240756    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:05.241511 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:05.241477    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:06.184935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:06.184898    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:06.185124 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:06.184963    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:06.237179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:06.237134    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:06.237476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:06.237447    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:07.184883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:07.184846    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:07.185277 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:07.184914    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:07.237380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:07.237348    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:07.237689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:07.237623    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:08.185004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:08.184963    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:27:08.185391 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:08.185044    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:27:08.238085 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:08.238048    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:27:08.238261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:08.238119    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 22:27:10.029992 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.029927    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:27:10.030801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.030766    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" podUID="c1020dfc-345f-42dd-953f-abae6b776f8d" containerName="maas-api" containerID="cri-o://6b86d33e657e984d6fbb5ed98eb7c2cc978eac8bd9ef0af90ba7c81e61c7a9d0" gracePeriod=30
Oct 30 22:27:10.065119 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.065085    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:10.066729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.066705    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:10.068032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.068011    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z"
Oct 30 22:27:10.090305 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.090230    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:10.093320 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.092327    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:10.093754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.093729    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj"
Oct 30 22:27:10.200323 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.200291    4295 generic.go:356] "Generic (PLEG): container finished" podID="c1020dfc-345f-42dd-953f-abae6b776f8d" containerID="6b86d33e657e984d6fbb5ed98eb7c2cc978eac8bd9ef0af90ba7c81e61c7a9d0" exitCode=0
Oct 30 22:27:10.200557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.200364    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" event={"ID":"c1020dfc-345f-42dd-953f-abae6b776f8d","Type":"ContainerDied","Data":"6b86d33e657e984d6fbb5ed98eb7c2cc978eac8bd9ef0af90ba7c81e61c7a9d0"}
Oct 30 22:27:10.272049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.271917    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:27:10.283237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.283122    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.378705 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.377603    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:27:10.391419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.391222    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:27:10.409918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.409877    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.409960    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410047    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410102    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s24c\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410209 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410140    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410222    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410274    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410315    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.410389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.410355    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.511266 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511233    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\" (UniqueName: \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\") pod \"c1020dfc-345f-42dd-953f-abae6b776f8d\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") "
Oct 30 22:27:10.511467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511328    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\") pod \"c1020dfc-345f-42dd-953f-abae6b776f8d\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") "
Oct 30 22:27:10.511467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511422    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\") pod \"c1020dfc-345f-42dd-953f-abae6b776f8d\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") "
Oct 30 22:27:10.511467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511457    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") pod \"c1020dfc-345f-42dd-953f-abae6b776f8d\" (UID: \"c1020dfc-345f-42dd-953f-abae6b776f8d\") "
Oct 30 22:27:10.511823 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511795    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.511931 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511899    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.511987 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511955    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.511999    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512042    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512090 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512065    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca" pod "c1020dfc-345f-42dd-953f-abae6b776f8d" (UID: "c1020dfc-345f-42dd-953f-abae6b776f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:10.512207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512148    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512226    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512315    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512390 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512356    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512445 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512384    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s24c\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.512499 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512460    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1020dfc-345f-42dd-953f-abae6b776f8d-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:10.513167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.512550    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.513634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.513613    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.513825 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.513630    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.513825 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.513747    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.514700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.514530    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs" pod "c1020dfc-345f-42dd-953f-abae6b776f8d" (UID: "c1020dfc-345f-42dd-953f-abae6b776f8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:10.514937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.514911    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d" pod "c1020dfc-345f-42dd-953f-abae6b776f8d" (UID: "c1020dfc-345f-42dd-953f-abae6b776f8d"). InnerVolumeSpecName "kube-api-access-9v72d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:10.516122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.516094    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.516237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.516147    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls" pod "c1020dfc-345f-42dd-953f-abae6b776f8d" (UID: "c1020dfc-345f-42dd-953f-abae6b776f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:10.516320 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.516298    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.578125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.578059    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.578334 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.578304    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s24c\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:10.614302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.614258    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:10.614302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.614292    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v72d\" (UniqueName: \"kubernetes.io/projected/c1020dfc-345f-42dd-953f-abae6b776f8d-kube-api-access-9v72d\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:10.614302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.614307    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1020dfc-345f-42dd-953f-abae6b776f8d-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:10.622385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:10.622348    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:11.107703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.070570    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:27:11.107703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.098133    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.107703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.098229    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.107703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.098274    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.206844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.206815    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"
Oct 30 22:27:11.206997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.206815    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb" event={"ID":"c1020dfc-345f-42dd-953f-abae6b776f8d","Type":"ContainerDied","Data":"3abf7e8f16328b0ad59c61d092487f0cef2cbb362e4e7d010e0b1daded81f14b"}
Oct 30 22:27:11.206997 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.206952    4295 scope.go:122] "RemoveContainer" containerID="6b86d33e657e984d6fbb5ed98eb7c2cc978eac8bd9ef0af90ba7c81e61c7a9d0"
Oct 30 22:27:11.208509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.208464    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" event={"ID":"f73e98a7-526e-4e71-9c0e-90d6f40fa00b","Type":"ContainerStarted","Data":"88b54c51d4d15d193c844794b3ac413e2683435ec8615930526713df35f00e4c"}
Oct 30 22:27:11.379655 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.379567    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:27:11.385419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.385390    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ff338b-786f5ff7c8-ztbgb"]
Oct 30 22:27:11.495396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:11.495365    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1020dfc-345f-42dd-953f-abae6b776f8d" path="/var/lib/kubelet/pods/c1020dfc-345f-42dd-953f-abae6b776f8d/volumes"
Oct 30 22:27:12.027957 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.027908    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main" containerID="cri-o://46d1374f588cc778044f867944d8c1929f0f3e4c8dfd262c68a1506ecf9f2cf4" gracePeriod=45
Oct 30 22:27:12.226830 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.226795    4295 generic.go:356] "Generic (PLEG): container finished" podID="1619a3eb-e341-416c-b31d-95d099322b62" containerID="46d1374f588cc778044f867944d8c1929f0f3e4c8dfd262c68a1506ecf9f2cf4" exitCode=0
Oct 30 22:27:12.227314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.226955    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" event={"ID":"1619a3eb-e341-416c-b31d-95d099322b62","Type":"ContainerDied","Data":"46d1374f588cc778044f867944d8c1929f0f3e4c8dfd262c68a1506ecf9f2cf4"}
Oct 30 22:27:12.229768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.229736    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" event={"ID":"f73e98a7-526e-4e71-9c0e-90d6f40fa00b","Type":"ContainerStarted","Data":"27cc1fb7889e235976ed564106f9442f3324638425862b7dffde1b6e978cd9f8"}
Oct 30 22:27:12.345726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.345699    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:27:12.355840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.355776    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podStartSLOduration=2.355759164 podStartE2EDuration="2.355759164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:12.352794791 +0000 UTC m=+2234.995388579" watchObservedRunningTime="2026-10-30 22:27:12.355759164 +0000 UTC m=+2234.998352955"
Oct 30 22:27:12.626099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.626007    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:12.649867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.649831    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:12.650782 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.650741    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652211    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\" (UniqueName: \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652260    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\" (UniqueName: \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652326    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652413    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652469    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.652619    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\") pod \"1619a3eb-e341-416c-b31d-95d099322b62\" (UID: \"1619a3eb-e341-416c-b31d-95d099322b62\") "
Oct 30 22:27:12.656025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.653273    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.657285 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.657253    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.657396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.657340    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.683728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.662343    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:12.683728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.663386    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.683728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.679176    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f" pod "1619a3eb-e341-416c-b31d-95d099322b62" (UID: "1619a3eb-e341-416c-b31d-95d099322b62"). InnerVolumeSpecName "kube-api-access-sss8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.757412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757377    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.757412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757411    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.757689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757424    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.757689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757436    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619a3eb-e341-416c-b31d-95d099322b62-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.757689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757446    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619a3eb-e341-416c-b31d-95d099322b62-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.757689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:12.757459    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss8f\" (UniqueName: \"kubernetes.io/projected/1619a3eb-e341-416c-b31d-95d099322b62-kube-api-access-sss8f\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:13.243690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.243342    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"
Oct 30 22:27:13.244200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.243718    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn" event={"ID":"1619a3eb-e341-416c-b31d-95d099322b62","Type":"ContainerDied","Data":"6865c0856a156007d8b02e6d38b0977fe933025eabde6ebbcb2a02d910f0cba7"}
Oct 30 22:27:13.244200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.243764    4295 scope.go:122] "RemoveContainer" containerID="46d1374f588cc778044f867944d8c1929f0f3e4c8dfd262c68a1506ecf9f2cf4"
Oct 30 22:27:13.428592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.420653    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:27:13.433345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.433307    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/premium-w0-6a0a09-kserve-8694c89769-58vgn"]
Oct 30 22:27:13.528738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.516913    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1619a3eb-e341-416c-b31d-95d099322b62" path="/var/lib/kubelet/pods/1619a3eb-e341-416c-b31d-95d099322b62/volumes"
Oct 30 22:27:13.625762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.624251    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:13.625762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:13.624314    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:14.625179 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:14.625144    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:14.625853 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:14.625802    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:15.024690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.024059    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"]
Oct 30 22:27:15.024909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.024780    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main"
Oct 30 22:27:15.024909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.024801    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main"
Oct 30 22:27:15.024909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.024841    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c1020dfc-345f-42dd-953f-abae6b776f8d" containerName="maas-api"
Oct 30 22:27:15.024909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.024850    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1020dfc-345f-42dd-953f-abae6b776f8d" containerName="maas-api"
Oct 30 22:27:15.025473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.025002    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1619a3eb-e341-416c-b31d-95d099322b62" containerName="main"
Oct 30 22:27:15.025473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.025025    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1020dfc-345f-42dd-953f-abae6b776f8d" containerName="maas-api"
Oct 30 22:27:15.076687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.076022    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.534689 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.514326    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"]
Oct 30 22:27:15.621089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.621055    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttj92\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-kube-api-access-ttj92\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.621526 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.621502    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622003    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622060    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622117    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622197    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622238    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622322    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.622408 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622375    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.645731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622909    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:15.645731 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.622959    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.727759    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.727863    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.727913    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.727993    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttj92\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-kube-api-access-ttj92\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.728035    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.728076    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.728112    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.728162    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.728686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.728247    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.743686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.730007    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.743686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.730304    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.743686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.730546    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.743686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.737063    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.743686 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.740878    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.756955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.756554    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.756955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.756854    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.828842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.824794    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttj92\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-kube-api-access-ttj92\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.832828 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.830175    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-2wc6x\" (UID: \"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:15.858289 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:15.858256    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:16.326376 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.326347    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"]
Oct 30 22:27:16.328456 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:27:16.328415    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf7d3d_a4f2_41d8_a647_08f8a80e87a4.slice/crio-8c524e3667266b3aeb5c70940f51eaff15da64681cbf7f455749279ab7921dc8 WatchSource:0}: Error finding container 8c524e3667266b3aeb5c70940f51eaff15da64681cbf7f455749279ab7921dc8: Status 404 returned error can't find the container with id 8c524e3667266b3aeb5c70940f51eaff15da64681cbf7f455749279ab7921dc8
Oct 30 22:27:16.331874 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.331834    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:16.332063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.331924    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:16.332063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.332005    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:16.586024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.585940    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" event={"ID":"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4","Type":"ContainerStarted","Data":"302cf8f4d1e16c0916ebc401c4adb4b08b932c3ea7a72c93bb126345f9393c53"}
Oct 30 22:27:16.586024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.585999    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" event={"ID":"f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4","Type":"ContainerStarted","Data":"8c524e3667266b3aeb5c70940f51eaff15da64681cbf7f455749279ab7921dc8"}
Oct 30 22:27:16.622727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.622694    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:16.622883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.622786    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:16.859240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:16.859152    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:17.623314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:17.623270    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:27:17.623534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:17.623333    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:27:17.860641 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:17.860606    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:17.861099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:17.860696    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:18.627831 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.627800    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:18.628302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.628275    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:18.629116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.629096    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:27:18.809964 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.809900    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podStartSLOduration=4.809879733 podStartE2EDuration="4.809879733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:16.716078496 +0000 UTC m=+2239.358672285" watchObservedRunningTime="2026-10-30 22:27:18.809879733 +0000 UTC m=+2241.452473522"
Oct 30 22:27:18.859124 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.859090    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:18.859288 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:18.859146    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:19.859297 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:19.859263    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:19.859711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:19.859317    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:20.858915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:20.858880    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:20.859097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:20.858932    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:21.721708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:21.721645    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:27:21.722207 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:21.721937    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" containerID="cri-o://6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da" gracePeriod=30
Oct 30 22:27:21.867094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:21.867054    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:21.867372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:21.867343    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:22.859443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:22.859404    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:22.859981 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:22.859467    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:23.607460 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:23.607423    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:27:23.858864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:23.858757    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:27:23.859044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:23.858842    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:27:24.535223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:24.535189    4295 patch_prober.go:29] interesting pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:24.535592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:24.535254    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:25.390770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.390734    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:25.391507 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.391469    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:25.392773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.392747    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x"
Oct 30 22:27:25.422204 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.422158    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:27:25.426235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.426214    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.502187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.502154    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:27:25.523842 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.523810    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbkm\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.523879    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524144    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524215    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524389 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524374    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524431    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524471    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524549 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524536    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.524593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.524569    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.625963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.625920    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.625971    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626090    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626151    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626190    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626217    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626242    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626317    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbkm\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626485 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626357    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626555    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626610    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626737    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626915    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.626963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.626932    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.628783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.628764    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.628871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.628795    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.692910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.692882    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.693073 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.693052    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbkm\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:25.740308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:25.740281    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:26.185873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.185844    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:27:26.186865 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:27:26.186840    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8c26da_f344_4c5f_a4a7_31d8577d6b0b.slice/crio-f12e9c4c3330951d00028e346459ef3d6226f515955353b704a0d791ad082f97 WatchSource:0}: Error finding container f12e9c4c3330951d00028e346459ef3d6226f515955353b704a0d791ad082f97: Status 404 returned error can't find the container with id f12e9c4c3330951d00028e346459ef3d6226f515955353b704a0d791ad082f97
Oct 30 22:27:26.189342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.189306    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:26.189424 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.189376    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:26.189424 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.189401    4295 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:26.685145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.685103    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" event={"ID":"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b","Type":"ContainerStarted","Data":"cb89ac822fc16422ad271f4158a7ce28766afde2ab2bfe7f31e60a22b67e5ec4"}
Oct 30 22:27:26.685556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.685172    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" event={"ID":"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b","Type":"ContainerStarted","Data":"f12e9c4c3330951d00028e346459ef3d6226f515955353b704a0d791ad082f97"}
Oct 30 22:27:26.741055 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:26.741031    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:27.025314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.025286    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:27:27.150758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.150702    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podStartSLOduration=2.150655678 podStartE2EDuration="2.150655678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:26.815099751 +0000 UTC m=+2249.457693542" watchObservedRunningTime="2026-10-30 22:27:27.150655678 +0000 UTC m=+2249.793249467"
Oct 30 22:27:27.155556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155529    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155570    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155627    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155681    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155760 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155702    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155827    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155856    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.155980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155937    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.156147 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.155983    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\") pod \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\" (UID: \"3e9350b3-2120-461c-9eb3-4144ed6f0a2f\") "
Oct 30 22:27:27.156211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156181    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:27.156402 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156352    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:27.156489 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156399    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:27.156547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156478    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:27.156547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156337    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:27.157058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156918    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.157058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156946    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.157058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156960    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.157058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156974    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.157058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.156986    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.158594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.158569    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:27.158744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.158726    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:27.158852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.158748    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "kube-api-access-74gjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:27.158897 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.158864    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy" pod "3e9350b3-2120-461c-9eb3-4144ed6f0a2f" (UID: "3e9350b3-2120-461c-9eb3-4144ed6f0a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:27.258617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.258530    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.258617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.258565    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74gjn\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-kube-api-access-74gjn\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.258617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.258576    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.258617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.258585    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9350b3-2120-461c-9eb3-4144ed6f0a2f-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:27.268508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.268476    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:27:27.716920 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.716885    4295 generic.go:356] "Generic (PLEG): container finished" podID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerID="6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da" exitCode=0
Oct 30 22:27:27.717322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.716940    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"
Oct 30 22:27:27.717322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.716975    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerDied","Data":"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"}
Oct 30 22:27:27.717322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.717013    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9" event={"ID":"3e9350b3-2120-461c-9eb3-4144ed6f0a2f","Type":"ContainerDied","Data":"2ce5a4153c82899189b22b5116a48d8e03ccd5fa817e239f25afd0eb2761f460"}
Oct 30 22:27:27.717322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.717033    4295 scope.go:122] "RemoveContainer" containerID="6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"
Oct 30 22:27:27.726844 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.726821    4295 scope.go:122] "RemoveContainer" containerID="82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"
Oct 30 22:27:27.735647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.735630    4295 scope.go:122] "RemoveContainer" containerID="6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"
Oct 30 22:27:27.735926 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:27.735897    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da\": container with ID starting with 6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da not found: ID does not exist" containerID="6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"
Oct 30 22:27:27.735978 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.735935    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da"} err="failed to get container status \"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da\": rpc error: code = NotFound desc = could not find container \"6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da\": container with ID starting with 6bb0f3af30431ee58f3f2d5029e3267eb824338bb53b1e339e2412f5831450da not found: ID does not exist"
Oct 30 22:27:27.735978 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.735954    4295 scope.go:122] "RemoveContainer" containerID="82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"
Oct 30 22:27:27.736216 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:27.736198    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8\": container with ID starting with 82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8 not found: ID does not exist" containerID="82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"
Oct 30 22:27:27.736267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.736223    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8"} err="failed to get container status \"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8\": rpc error: code = NotFound desc = could not find container \"82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8\": container with ID starting with 82e6fbe79e232355c855b319644f9939e253711833376f63d4698b4ee6f1fcb8 not found: ID does not exist"
Oct 30 22:27:27.742909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.742888    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:27:27.742983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.742929    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:27:27.887169 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.887085    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:27:27.913676 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:27.913637    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9"]
Oct 30 22:27:28.740717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:28.740651    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:27:28.741104 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:28.740729    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:27:29.516813 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:29.516773    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" path="/var/lib/kubelet/pods/3e9350b3-2120-461c-9eb3-4144ed6f0a2f/volumes"
Oct 30 22:27:29.741482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:29.741448    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:27:29.741917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:29.741511    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:27:30.741148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:30.741111    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:27:30.741333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:30.741175    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:27:31.741062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:31.741032    4295 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:27:31.741494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:31.741094    4295 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:27:32.745858 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:32.745826    4295 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:32.746254 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:32.746100    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:32.746768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:32.746749    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:27:33.342619 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:33.342586    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:27:38.343711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:38.343643    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing" containerID="cri-o://7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb" gracePeriod=25
Oct 30 22:27:41.010581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:41.010541    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:42.289447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.289405    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" containerID="cri-o://fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d" gracePeriod=45
Oct 30 22:27:42.313280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.313245    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 22:27:42.571986 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.571953    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:27:42.598907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.598872    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\" (UniqueName: \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.598924    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\" (UniqueName: \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.598994    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599055    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599170    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599230    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\") pod \"157cd977-670d-41d5-af48-0b9005280f57\" (UID: \"157cd977-670d-41d5-af48-0b9005280f57\") "
Oct 30 22:27:42.599373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599300    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.599426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599374    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.599489 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599450    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.599721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599703    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.599721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599723    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.600052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.599732    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.601420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.601397    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:42.601420 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.601401    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.601548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.601455    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq" pod "157cd977-670d-41d5-af48-0b9005280f57" (UID: "157cd977-670d-41d5-af48-0b9005280f57"). InnerVolumeSpecName "kube-api-access-n8jjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:42.701140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.701102    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157cd977-670d-41d5-af48-0b9005280f57-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.701140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.701137    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8jjq\" (UniqueName: \"kubernetes.io/projected/157cd977-670d-41d5-af48-0b9005280f57-kube-api-access-n8jjq\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.701140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.701150    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157cd977-670d-41d5-af48-0b9005280f57-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.833809 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.833715    4295 generic.go:356] "Generic (PLEG): container finished" podID="157cd977-670d-41d5-af48-0b9005280f57" containerID="fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d" exitCode=0
Oct 30 22:27:42.833809 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.833785    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"
Oct 30 22:27:42.834001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.833809    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" event={"ID":"157cd977-670d-41d5-af48-0b9005280f57","Type":"ContainerDied","Data":"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"}
Oct 30 22:27:42.834001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.833857    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6" event={"ID":"157cd977-670d-41d5-af48-0b9005280f57","Type":"ContainerDied","Data":"cec61e348bf2e317310cae26f89fa95c264258e9596c64bf4db01de26dd81e3d"}
Oct 30 22:27:42.834001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.833877    4295 scope.go:122] "RemoveContainer" containerID="fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"
Oct 30 22:27:42.847395 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.847368    4295 scope.go:122] "RemoveContainer" containerID="fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"
Oct 30 22:27:42.860054 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:42.860018    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d\": container with ID starting with fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d not found: ID does not exist" containerID="fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"
Oct 30 22:27:42.860217 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:42.860067    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d"} err="failed to get container status \"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d\": rpc error: code = NotFound desc = could not find container \"fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d\": container with ID starting with fb02eb367f51d1e88dde9e7d3a05394dd7e9bc2fd00baed5c42a1eb0180dd53d not found: ID does not exist"
Oct 30 22:27:43.035954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:43.035920    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:27:43.042287 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:43.042257    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/premium-w1-9cff60-kserve-558d6dc6dc-vbvm6"]
Oct 30 22:27:43.492983 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:43.492941    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157cd977-670d-41d5-af48-0b9005280f57" path="/var/lib/kubelet/pods/157cd977-670d-41d5-af48-0b9005280f57/volumes"
Oct 30 22:27:46.756025 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:46.755980    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:27:51.010793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:51.010751    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:53.608232 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:53.608194    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:27:58.612623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.612595    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:27:58.713806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.713764    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\") pod \"680bf5c9-2b25-407f-8a86-896031ab5184\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") "
Oct 30 22:27:58.713806 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.713803    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\" (UniqueName: \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\") pod \"680bf5c9-2b25-407f-8a86-896031ab5184\" (UID: \"680bf5c9-2b25-407f-8a86-896031ab5184\") "
Oct 30 22:27:58.714223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.714196    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume" pod "680bf5c9-2b25-407f-8a86-896031ab5184" (UID: "680bf5c9-2b25-407f-8a86-896031ab5184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:58.716062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.716040    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w" pod "680bf5c9-2b25-407f-8a86-896031ab5184" (UID: "680bf5c9-2b25-407f-8a86-896031ab5184"). InnerVolumeSpecName "kube-api-access-lf57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:58.815988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.815905    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680bf5c9-2b25-407f-8a86-896031ab5184-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.815988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.815936    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf57w\" (UniqueName: \"kubernetes.io/projected/680bf5c9-2b25-407f-8a86-896031ab5184-kube-api-access-lf57w\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.923834 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.923799    4295 generic.go:356] "Generic (PLEG): container finished" podID="680bf5c9-2b25-407f-8a86-896031ab5184" containerID="7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb" exitCode=0
Oct 30 22:27:58.924041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.923849    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"
Oct 30 22:27:58.924041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.923884    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" event={"ID":"680bf5c9-2b25-407f-8a86-896031ab5184","Type":"ContainerDied","Data":"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"}
Oct 30 22:27:58.924041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.923942    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm" event={"ID":"680bf5c9-2b25-407f-8a86-896031ab5184","Type":"ContainerDied","Data":"fef79d49a6eedf266893ced850f2fb68f3555c2b1d224b9a356e36e1865bfaa7"}
Oct 30 22:27:58.924041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.923974    4295 scope.go:122] "RemoveContainer" containerID="7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"
Oct 30 22:27:58.937702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.937650    4295 scope.go:122] "RemoveContainer" containerID="7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"
Oct 30 22:27:58.938136 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:27:58.938109    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb\": container with ID starting with 7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb not found: ID does not exist" containerID="7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"
Oct 30 22:27:58.938241 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:58.938147    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb"} err="failed to get container status \"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb\": rpc error: code = NotFound desc = could not find container \"7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb\": container with ID starting with 7166c1e08b10e79d496bf8c9b8ba8b17806b2bdd2fe7a9c8be3eb4d2c9ad7feb not found: ID does not exist"
Oct 30 22:27:59.093903 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:59.093822    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:27:59.100321 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:59.100289    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm"]
Oct 30 22:27:59.493044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:27:59.493011    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" path="/var/lib/kubelet/pods/680bf5c9-2b25-407f-8a86-896031ab5184/volumes"
Oct 30 22:28:01.784311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:01.784272    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" podUID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" containerName="main" containerID="cri-o://bd96da5691711ae6bada8ec71333f11258bbd6a7404f2afe5a1ae2712706cae8" gracePeriod=45
Oct 30 22:28:01.943566 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:01.943497    4295 generic.go:356] "Generic (PLEG): container finished" podID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" containerID="bd96da5691711ae6bada8ec71333f11258bbd6a7404f2afe5a1ae2712706cae8" exitCode=0
Oct 30 22:28:01.943748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:01.943564    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" event={"ID":"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7","Type":"ContainerDied","Data":"bd96da5691711ae6bada8ec71333f11258bbd6a7404f2afe5a1ae2712706cae8"}
Oct 30 22:28:02.050009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.049942    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:28:02.155538 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155507    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\" (UniqueName: \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.155736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155545    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.155736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155586    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.155866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155733    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.155866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155847    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.155866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155847    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:02.155866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155861    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:02.156077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.155938    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\") pod \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\" (UID: \"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7\") "
Oct 30 22:28:02.156131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.156084    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:02.156521 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.156463    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.156521 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.156484    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.156521 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.156493    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.158046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.158023    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:02.158142 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.158052    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:02.158142 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.158115    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp" pod "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" (UID: "2f499cce-8561-4ff6-b2f7-dd83e9dc00f7"). InnerVolumeSpecName "kube-api-access-65cwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:02.257582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.257545    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65cwp\" (UniqueName: \"kubernetes.io/projected/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-kube-api-access-65cwp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.257582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.257578    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.257582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.257588    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:02.949737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.949703    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"
Oct 30 22:28:02.949737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.949717    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp" event={"ID":"2f499cce-8561-4ff6-b2f7-dd83e9dc00f7","Type":"ContainerDied","Data":"29ae1b5fc3c4c86c29384c533166ff69a9f4c591580e4ecdfad95d426b6b9d4d"}
Oct 30 22:28:02.950239 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:02.949769    4295 scope.go:122] "RemoveContainer" containerID="bd96da5691711ae6bada8ec71333f11258bbd6a7404f2afe5a1ae2712706cae8"
Oct 30 22:28:03.119475 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:03.119439    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:28:03.169722 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:03.169692    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-422c1029/e2e-lifecycle-model-422c1029-kserve-76c6b89d5-jzdfp"]
Oct 30 22:28:03.493317 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:03.493285    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" path="/var/lib/kubelet/pods/2f499cce-8561-4ff6-b2f7-dd83e9dc00f7/volumes"
Oct 30 22:28:23.716743 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:28:23.715900    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:28:24.680694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:24.680637    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:28:32.197815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:32.197774    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:28:32.198414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:32.198027    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy" containerID="cri-o://9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8" gracePeriod=30
Oct 30 22:28:37.467206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.467150    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:28:37.545346 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545312    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545412    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545453    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545499    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545527    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545560    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545606    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545714 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545706    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545933 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545793    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\") pod \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\" (UID: \"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1\") "
Oct 30 22:28:37.545933 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545856    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:37.545933 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.545864    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:37.546148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546007    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:37.546218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546175    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546349    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546492    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546514    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546529    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546543    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.546700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.546559    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.547802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.547779    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:37.548074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.548045    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "kube-api-access-2dwgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:37.548229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.548213    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:37.548279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.548221    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo" pod "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" (UID: "f942fc11-7561-4f06-89bc-1c7cd3a3bdd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:37.647573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.647536    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.647573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.647571    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.647791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.647587    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:37.647791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:37.647599    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwgs\" (UniqueName: \"kubernetes.io/projected/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1-kube-api-access-2dwgs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:38.255050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.255012    4295 generic.go:356] "Generic (PLEG): container finished" podID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerID="9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8" exitCode=0
Oct 30 22:28:38.255251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.255074    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" event={"ID":"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1","Type":"ContainerDied","Data":"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"}
Oct 30 22:28:38.255251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.255089    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"
Oct 30 22:28:38.255251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.255139    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx" event={"ID":"f942fc11-7561-4f06-89bc-1c7cd3a3bdd1","Type":"ContainerDied","Data":"c8f795a838b50764a59de48c6a2e743f8b9ad069fc845dcaf288e9651c9df39b"}
Oct 30 22:28:38.255251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.255159    4295 scope.go:122] "RemoveContainer" containerID="9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"
Oct 30 22:28:38.265383 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.265363    4295 scope.go:122] "RemoveContainer" containerID="9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"
Oct 30 22:28:38.265622 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:28:38.265602    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8\": container with ID starting with 9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8 not found: ID does not exist" containerID="9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"
Oct 30 22:28:38.265708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.265634    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8"} err="failed to get container status \"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8\": rpc error: code = NotFound desc = could not find container \"9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8\": container with ID starting with 9a9feb82ab3261b579c9c722b015d5ce8b84a9a7f3e67eeb01f304c9a528fae8 not found: ID does not exist"
Oct 30 22:28:38.474970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.474934    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:28:38.525438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:38.525364    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx"]
Oct 30 22:28:39.493318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:39.493276    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" path="/var/lib/kubelet/pods/f942fc11-7561-4f06-89bc-1c7cd3a3bdd1/volumes"
Oct 30 22:28:39.770732 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:39.770617    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main" containerID="cri-o://386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df" gracePeriod=45
Oct 30 22:28:40.021907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.021842    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:28:40.072396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072365    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072577 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072405    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072577 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072446    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\" (UniqueName: \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072623    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072709    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:40.072837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072720    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072815    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\") pod \"17e2a475-5742-4f29-9572-e8d2ec726ce2\" (UID: \"17e2a475-5742-4f29-9572-e8d2ec726ce2\") "
Oct 30 22:28:40.072953 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.072914    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:40.073279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.073054    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:40.073279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.073253    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.073279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.073275    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.074810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.074778    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:40.074918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.074842    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "kube-api-access-x889b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:40.074918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.074851    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs" pod "17e2a475-5742-4f29-9572-e8d2ec726ce2" (UID: "17e2a475-5742-4f29-9572-e8d2ec726ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:40.174851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.174811    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.174851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.174837    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e2a475-5742-4f29-9572-e8d2ec726ce2-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.174851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.174847    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x889b\" (UniqueName: \"kubernetes.io/projected/17e2a475-5742-4f29-9572-e8d2ec726ce2-kube-api-access-x889b\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.174851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.174858    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e2a475-5742-4f29-9572-e8d2ec726ce2-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.268257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.268224    4295 generic.go:356] "Generic (PLEG): container finished" podID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerID="386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df" exitCode=0
Oct 30 22:28:40.268450 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.268276    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"
Oct 30 22:28:40.268450 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.268308    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" event={"ID":"17e2a475-5742-4f29-9572-e8d2ec726ce2","Type":"ContainerDied","Data":"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"}
Oct 30 22:28:40.268450 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.268350    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62" event={"ID":"17e2a475-5742-4f29-9572-e8d2ec726ce2","Type":"ContainerDied","Data":"00f1842bfa3a6a5eb833c350c98ea8fb08c90f32debaa289785168c0328a33c8"}
Oct 30 22:28:40.268450 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.268369    4295 scope.go:122] "RemoveContainer" containerID="386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"
Oct 30 22:28:40.281336 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.281315    4295 scope.go:122] "RemoveContainer" containerID="386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"
Oct 30 22:28:40.281598 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:28:40.281577    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df\": container with ID starting with 386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df not found: ID does not exist" containerID="386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"
Oct 30 22:28:40.281674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.281608    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df"} err="failed to get container status \"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df\": rpc error: code = NotFound desc = could not find container \"386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df\": container with ID starting with 386b48a73fe56a2dedb91101c53acf06c5a0124f6492b8be94bc3ddbcaa362df not found: ID does not exist"
Oct 30 22:28:40.439990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.439958    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:28:40.446474 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:40.446445    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/premium-w2-d8837a-kserve-5b647787c5-m2f62"]
Oct 30 22:28:41.494248 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:41.494211    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" path="/var/lib/kubelet/pods/17e2a475-5742-4f29-9572-e8d2ec726ce2/volumes"
Oct 30 22:28:42.041325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.041292    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:28:42.041582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.041558    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy" containerID="cri-o://8b6c2af9002394893f76e0677b97ed277020d7c5cf859426ca7431ea67f2210e" gracePeriod=30
Oct 30 22:28:42.070852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.070820    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:28:42.071041 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.071021    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy" containerID="cri-o://27cc1fb7889e235976ed564106f9442f3324638425862b7dffde1b6e978cd9f8" gracePeriod=30
Oct 30 22:28:42.081925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.081896    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:28:42.082128 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.082107    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy" containerID="cri-o://c33aeb3df5db1973dbcc4140cba784ce4c2a5d6f9a38763c8f4a7dccef85de25" gracePeriod=30
Oct 30 22:28:42.091412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.091387    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:28:42.091574 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:42.091557    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy" containerID="cri-o://cb89ac822fc16422ad271f4158a7ce28766afde2ab2bfe7f31e60a22b67e5ec4" gracePeriod=30
Oct 30 22:28:47.308325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.308073    4295 generic.go:356] "Generic (PLEG): container finished" podID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerID="27cc1fb7889e235976ed564106f9442f3324638425862b7dffde1b6e978cd9f8" exitCode=0
Oct 30 22:28:47.308325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.308210    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" event={"ID":"f73e98a7-526e-4e71-9c0e-90d6f40fa00b","Type":"ContainerDied","Data":"27cc1fb7889e235976ed564106f9442f3324638425862b7dffde1b6e978cd9f8"}
Oct 30 22:28:47.310791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.310622    4295 generic.go:356] "Generic (PLEG): container finished" podID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerID="cb89ac822fc16422ad271f4158a7ce28766afde2ab2bfe7f31e60a22b67e5ec4" exitCode=0
Oct 30 22:28:47.310791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.310732    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" event={"ID":"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b","Type":"ContainerDied","Data":"cb89ac822fc16422ad271f4158a7ce28766afde2ab2bfe7f31e60a22b67e5ec4"}
Oct 30 22:28:47.312453 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.312271    4295 generic.go:356] "Generic (PLEG): container finished" podID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerID="c33aeb3df5db1973dbcc4140cba784ce4c2a5d6f9a38763c8f4a7dccef85de25" exitCode=0
Oct 30 22:28:47.312453 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.312407    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" event={"ID":"989cbd04-3b28-461a-9448-a2736c8f7aa2","Type":"ContainerDied","Data":"c33aeb3df5db1973dbcc4140cba784ce4c2a5d6f9a38763c8f4a7dccef85de25"}
Oct 30 22:28:47.314576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.314391    4295 generic.go:356] "Generic (PLEG): container finished" podID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerID="8b6c2af9002394893f76e0677b97ed277020d7c5cf859426ca7431ea67f2210e" exitCode=0
Oct 30 22:28:47.314576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.314453    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerDied","Data":"8b6c2af9002394893f76e0677b97ed277020d7c5cf859426ca7431ea67f2210e"}
Oct 30 22:28:47.314576 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.314486    4295 scope.go:122] "RemoveContainer" containerID="55fe89166a1afe1ca22857b31673892c6a64231e72d7199dbd77444efa983f60"
Oct 30 22:28:47.386358 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.386329    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:28:47.389984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.389958    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:28:47.431937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.431910    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:28:47.438763 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.438745    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:28:47.477954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.477926    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.477968    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478072 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478063    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478160    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478169    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478222    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478283    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478309    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478315    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478396    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478405    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478447    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478457    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478478    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478551    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478585 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478582    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478628    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478821    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478873    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478885    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478905    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478937    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.478970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.478969    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.479318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479018    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\") pod \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\" (UID: \"f73e98a7-526e-4e71-9c0e-90d6f40fa00b\") "
Oct 30 22:28:47.479318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479053    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.479318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479121    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\") pod \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\" (UID: \"cd26d361-a47d-4531-a0e5-74ee9b9b7a05\") "
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479621    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479643    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479690    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479704    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479719    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.479737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479733    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.480121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479746    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.480121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479759    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.480121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479713    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.480121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.479761    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.481035 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.481006    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.481612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.481572    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "kube-api-access-6s24c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.481792 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.481753    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.481915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.481805    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.481915 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.481847    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.482162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.482137    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy" pod "f73e98a7-526e-4e71-9c0e-90d6f40fa00b" (UID: "f73e98a7-526e-4e71-9c0e-90d6f40fa00b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.482737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.482719    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.482891 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.482875    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh" pod "cd26d361-a47d-4531-a0e5-74ee9b9b7a05" (UID: "cd26d361-a47d-4531-a0e5-74ee9b9b7a05"). InnerVolumeSpecName "kube-api-access-j8wbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.581133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581051    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581118    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581370 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581147    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581497    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581539    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581573    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.581640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581606    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.581854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581675    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.581854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581723    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.581854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581774    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.581854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581818    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.582050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581886    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.582050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.581969    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.582050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582007    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.582050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582042    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.582249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582085    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.582249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582145    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\") pod \"989cbd04-3b28-461a-9448-a2736c8f7aa2\" (UID: \"989cbd04-3b28-461a-9448-a2736c8f7aa2\") "
Oct 30 22:28:47.582249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582207    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\") pod \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\" (UID: \"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b\") "
Oct 30 22:28:47.582749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582725    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582753    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582769    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582783    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s24c\" (UniqueName: \"kubernetes.io/projected/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-kube-api-access-6s24c\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582798    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582812    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582825    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.582849 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582839    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e98a7-526e-4e71-9c0e-90d6f40fa00b-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.583224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582852    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.583224 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.582867    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/cd26d361-a47d-4531-a0e5-74ee9b9b7a05-kube-api-access-j8wbh\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.584032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584003    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.584314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584285    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.584467 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584440    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.584591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584566    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.584839 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584816    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.584927 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.584846    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.585081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585058    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.585081 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585069    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.585225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585199    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.585401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585375    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.585509 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585414    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.585818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.585694    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.586908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.586886    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.588193 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.588146    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.588290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.588262    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "kube-api-access-tsznt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.588734 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.588713    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.588818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.588741    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token" pod "989cbd04-3b28-461a-9448-a2736c8f7aa2" (UID: "989cbd04-3b28-461a-9448-a2736c8f7aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.589237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.589212    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm" pod "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" (UID: "ba8c26da-f344-4c5f-a4a7-31d8577d6b0b"). InnerVolumeSpecName "kube-api-access-8wbkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.683721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683691    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.683721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683714    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsznt\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-kube-api-access-tsznt\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.683721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683725    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683735    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683744    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683752    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683760    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683769    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683776    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683785    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683793    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683800    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683810    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbkm\" (UniqueName: \"kubernetes.io/projected/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-kube-api-access-8wbkm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683818    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683827    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989cbd04-3b28-461a-9448-a2736c8f7aa2-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683836    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989cbd04-3b28-461a-9448-a2736c8f7aa2-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683844    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989cbd04-3b28-461a-9448-a2736c8f7aa2-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.684010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:47.683852    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:48.322201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.322118    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"
Oct 30 22:28:48.322201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.322155    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm" event={"ID":"cd26d361-a47d-4531-a0e5-74ee9b9b7a05","Type":"ContainerDied","Data":"9340e6bdea2704ba098fb7dff5f3c0c7713d30f1c63ee131568bb33894e02687"}
Oct 30 22:28:48.322776 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.322227    4295 scope.go:122] "RemoveContainer" containerID="8b6c2af9002394893f76e0677b97ed277020d7c5cf859426ca7431ea67f2210e"
Oct 30 22:28:48.324788 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.324718    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"
Oct 30 22:28:48.324788 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.324753    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl" event={"ID":"f73e98a7-526e-4e71-9c0e-90d6f40fa00b","Type":"ContainerDied","Data":"88b54c51d4d15d193c844794b3ac413e2683435ec8615930526713df35f00e4c"}
Oct 30 22:28:48.326533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.326514    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"
Oct 30 22:28:48.326621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.326538    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q" event={"ID":"ba8c26da-f344-4c5f-a4a7-31d8577d6b0b","Type":"ContainerDied","Data":"f12e9c4c3330951d00028e346459ef3d6226f515955353b704a0d791ad082f97"}
Oct 30 22:28:48.328362 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.328340    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"
Oct 30 22:28:48.328464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.328359    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v" event={"ID":"989cbd04-3b28-461a-9448-a2736c8f7aa2","Type":"ContainerDied","Data":"135197432af4f0068bc5e96c6489a624305c1868e7b434b61c61dbe9cbf02a51"}
Oct 30 22:28:48.333397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.333376    4295 scope.go:122] "RemoveContainer" containerID="27cc1fb7889e235976ed564106f9442f3324638425862b7dffde1b6e978cd9f8"
Oct 30 22:28:48.343162 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.343147    4295 scope.go:122] "RemoveContainer" containerID="cb89ac822fc16422ad271f4158a7ce28766afde2ab2bfe7f31e60a22b67e5ec4"
Oct 30 22:28:48.351345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.351329    4295 scope.go:122] "RemoveContainer" containerID="c33aeb3df5db1973dbcc4140cba784ce4c2a5d6f9a38763c8f4a7dccef85de25"
Oct 30 22:28:48.493787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.493756    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:28:48.543198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.543162    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q"]
Oct 30 22:28:48.657887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.657812    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:28:48.665323 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.665296    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm"]
Oct 30 22:28:48.779433 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.779402    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:28:48.785400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.785373    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl"]
Oct 30 22:28:48.900994 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.900966    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:28:48.910773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:48.910699    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v"]
Oct 30 22:28:49.493679 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:49.493630    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" path="/var/lib/kubelet/pods/989cbd04-3b28-461a-9448-a2736c8f7aa2/volumes"
Oct 30 22:28:49.494118 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:49.494103    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" path="/var/lib/kubelet/pods/ba8c26da-f344-4c5f-a4a7-31d8577d6b0b/volumes"
Oct 30 22:28:49.494505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:49.494493    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" path="/var/lib/kubelet/pods/cd26d361-a47d-4531-a0e5-74ee9b9b7a05/volumes"
Oct 30 22:28:49.495225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:49.495211    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" path="/var/lib/kubelet/pods/f73e98a7-526e-4e71-9c0e-90d6f40fa00b/volumes"
Oct 30 22:28:53.610374 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:28:53.610330    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:28:56.115707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.114752    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-5l25z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:28:56.115707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.114832    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z" podUID="7a0403e8-735c-4807-92d8-7a0fed332e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:56.156700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.147827    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:28:56.156700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.147948    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:56.448364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.446358    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:28:56.448364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:28:56.446423    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x" podUID="f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:23.607883 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:29:23.607834    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:29:25.060028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.059995    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 22:29:25.060459 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.060202    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" podUID="b3247850-8f20-4132-8fd7-3cdc84e454ad" containerName="maas-api" containerID="cri-o://08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468" gracePeriod=30
Oct 30 22:29:25.341775 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.341752    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 22:29:25.489797 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.489762    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") pod \"b3247850-8f20-4132-8fd7-3cdc84e454ad\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") "
Oct 30 22:29:25.492302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.490353    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\" (UniqueName: \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\") pod \"b3247850-8f20-4132-8fd7-3cdc84e454ad\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") "
Oct 30 22:29:25.492302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.490466    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\") pod \"b3247850-8f20-4132-8fd7-3cdc84e454ad\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") "
Oct 30 22:29:25.492302 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.490555    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\") pod \"b3247850-8f20-4132-8fd7-3cdc84e454ad\" (UID: \"b3247850-8f20-4132-8fd7-3cdc84e454ad\") "
Oct 30 22:29:25.493280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.493153    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca" pod "b3247850-8f20-4132-8fd7-3cdc84e454ad" (UID: "b3247850-8f20-4132-8fd7-3cdc84e454ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:25.495012 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.494861    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs" pod "b3247850-8f20-4132-8fd7-3cdc84e454ad" (UID: "b3247850-8f20-4132-8fd7-3cdc84e454ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:25.500272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.500224    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q" pod "b3247850-8f20-4132-8fd7-3cdc84e454ad" (UID: "b3247850-8f20-4132-8fd7-3cdc84e454ad"). InnerVolumeSpecName "kube-api-access-2g82q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:25.502291 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.502260    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls" pod "b3247850-8f20-4132-8fd7-3cdc84e454ad" (UID: "b3247850-8f20-4132-8fd7-3cdc84e454ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:25.549260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.549228    4295 generic.go:356] "Generic (PLEG): container finished" podID="b3247850-8f20-4132-8fd7-3cdc84e454ad" containerID="08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468" exitCode=0
Oct 30 22:29:25.549361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.549276    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"
Oct 30 22:29:25.549409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.549277    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" event={"ID":"b3247850-8f20-4132-8fd7-3cdc84e454ad","Type":"ContainerDied","Data":"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"}
Oct 30 22:29:25.549409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.549400    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl" event={"ID":"b3247850-8f20-4132-8fd7-3cdc84e454ad","Type":"ContainerDied","Data":"159f7ccbb0dcd06b6560f6dd41ce73bc374ef9e81906bf204da4d0f2cd16e60c"}
Oct 30 22:29:25.549478 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.549426    4295 scope.go:122] "RemoveContainer" containerID="08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"
Oct 30 22:29:25.558638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.558619    4295 scope.go:122] "RemoveContainer" containerID="08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"
Oct 30 22:29:25.558922 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:29:25.558902    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468\": container with ID starting with 08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468 not found: ID does not exist" containerID="08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"
Oct 30 22:29:25.558991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.558934    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468"} err="failed to get container status \"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468\": rpc error: code = NotFound desc = could not find container \"08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468\": container with ID starting with 08eab2d5d0f57b209aae9c19ad49f97f5aa242aaa1c20709c9f3f6d47e27a468 not found: ID does not exist"
Oct 30 22:29:25.612979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.612819    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:25.612979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.612858    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g82q\" (UniqueName: \"kubernetes.io/projected/b3247850-8f20-4132-8fd7-3cdc84e454ad-kube-api-access-2g82q\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:25.612979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.612876    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3247850-8f20-4132-8fd7-3cdc84e454ad-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:25.612979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.612889    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3247850-8f20-4132-8fd7-3cdc84e454ad-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:25.718094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.718060    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 22:29:25.727681 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:25.727629    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d8837a-5c4b95b546-hjmtl"]
Oct 30 22:29:27.493474 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:27.493443    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3247850-8f20-4132-8fd7-3cdc84e454ad" path="/var/lib/kubelet/pods/b3247850-8f20-4132-8fd7-3cdc84e454ad/volumes"
Oct 30 22:29:40.634970 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.634899    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"]
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635286    4295 cpu_manager.go:414] "containerMap: removing container" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635298    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635310    4295 cpu_manager.go:414] "containerMap: removing container" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635315    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635325    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635329    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635338    4295 cpu_manager.go:414] "containerMap: removing container" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635343    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635351    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b3247850-8f20-4132-8fd7-3cdc84e454ad" containerName="maas-api"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635356    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3247850-8f20-4132-8fd7-3cdc84e454ad" containerName="maas-api"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635365    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635369    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635378    4295 cpu_manager.go:414] "containerMap: removing container" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635382    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635395    4295 cpu_manager.go:414] "containerMap: removing container" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy"
Oct 30 22:29:40.635399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635399    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635409    4295 cpu_manager.go:414] "containerMap: removing container" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635415    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635421    4295 cpu_manager.go:414] "containerMap: removing container" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635425    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635435    4295 cpu_manager.go:414] "containerMap: removing container" podUID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635439    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635490    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635498    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="157cd977-670d-41d5-af48-0b9005280f57" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635504    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3247850-8f20-4132-8fd7-3cdc84e454ad" containerName="maas-api"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635509    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f942fc11-7561-4f06-89bc-1c7cd3a3bdd1" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635514    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8c26da-f344-4c5f-a4a7-31d8577d6b0b" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635520    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73e98a7-526e-4e71-9c0e-90d6f40fa00b" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635526    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635531    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="989cbd04-3b28-461a-9448-a2736c8f7aa2" containerName="istio-proxy"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635535    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e2a475-5742-4f29-9572-e8d2ec726ce2" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635541    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f499cce-8561-4ff6-b2f7-dd83e9dc00f7" containerName="main"
Oct 30 22:29:40.635894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.635546    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="680bf5c9-2b25-407f-8a86-896031ab5184" containerName="payload-processing"
Oct 30 22:29:40.639999 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.639980    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.677591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.677562    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.677757 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.677640    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl4w\" (UniqueName: \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.677838 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.677799    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.705948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.705918    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"]
Oct 30 22:29:40.752489 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.752458    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:40.752640 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.752516    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:29:40.779010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.778979    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.779176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.779062    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl4w\" (UniqueName: \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.779176 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.779097    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.779724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.779647    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.781572 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.781551    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.842521 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.842489    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl4w\" (UniqueName: \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\") pod \"maas-api-revoke-sub-70407806-nkl8v\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:40.953338 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:40.953299    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:41.331638 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:29:41.331607    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b823cba_638a_4058_9635_fa8604d96c87.slice/crio-7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f WatchSource:0}: Error finding container 7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f: Status 404 returned error can't find the container with id 7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f
Oct 30 22:29:41.372399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:41.372372    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"]
Oct 30 22:29:41.630708 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:41.630670    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v" event={"ID":"5b823cba-638a-4058-9635-fa8604d96c87","Type":"ContainerStarted","Data":"7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f"}
Oct 30 22:29:41.839895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:41.839862    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:29:41.847605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:41.847577    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:29:42.637514 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:42.637439    4295 generic.go:356] "Generic (PLEG): container finished" podID="5b823cba-638a-4058-9635-fa8604d96c87" containerID="65fea916606834e006106cb02e1a4ed076e0192a68ffdbda6f6be4a5211184e1" exitCode=0
Oct 30 22:29:42.637680 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:42.637516    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v" event={"ID":"5b823cba-638a-4058-9635-fa8604d96c87","Type":"ContainerDied","Data":"65fea916606834e006106cb02e1a4ed076e0192a68ffdbda6f6be4a5211184e1"}
Oct 30 22:29:43.786787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.786759    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:43.811754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.811726    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\") pod \"5b823cba-638a-4058-9635-fa8604d96c87\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") "
Oct 30 22:29:43.811863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.811815    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\") pod \"5b823cba-638a-4058-9635-fa8604d96c87\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") "
Oct 30 22:29:43.811917 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.811903    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\" (UniqueName: \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\") pod \"5b823cba-638a-4058-9635-fa8604d96c87\" (UID: \"5b823cba-638a-4058-9635-fa8604d96c87\") "
Oct 30 22:29:43.812200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.812172    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca" pod "5b823cba-638a-4058-9635-fa8604d96c87" (UID: "5b823cba-638a-4058-9635-fa8604d96c87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:43.814010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.813981    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca" pod "5b823cba-638a-4058-9635-fa8604d96c87" (UID: "5b823cba-638a-4058-9635-fa8604d96c87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:43.814103 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.814082    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w" pod "5b823cba-638a-4058-9635-fa8604d96c87" (UID: "5b823cba-638a-4058-9635-fa8604d96c87"). InnerVolumeSpecName "kube-api-access-mvl4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:43.913044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.912971    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl4w\" (UniqueName: \"kubernetes.io/projected/5b823cba-638a-4058-9635-fa8604d96c87-kube-api-access-mvl4w\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:43.913044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.912997    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:43.913044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:43.913007    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b823cba-638a-4058-9635-fa8604d96c87-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:44.649815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:44.649786    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"
Oct 30 22:29:44.649815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:44.649789    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v" event={"ID":"5b823cba-638a-4058-9635-fa8604d96c87","Type":"ContainerDied","Data":"7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f"}
Oct 30 22:29:44.650050 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:44.649851    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f81431eb394099733ffd193ef9e45e84fea98e58263c9ceaf66aae9378dea5f"
Oct 30 22:29:47.833404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:47.833367    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:29:47.947379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:47.947344    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:29:52.834418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:52.834372    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing" containerID="cri-o://e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400" gracePeriod=25
Oct 30 22:29:52.948481 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:52.948416    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing" containerID="cri-o://a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c" gracePeriod=25
Oct 30 22:29:53.608138 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:29:53.608098    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:29:56.309921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:56.309875    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:29:56.360290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:56.360259    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:29:56.860833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:56.860792    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main" containerID="cri-o://45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7" gracePeriod=45
Oct 30 22:29:56.866949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:56.866917    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main" containerID="cri-o://ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac" gracePeriod=45
Oct 30 22:29:57.128315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.128290    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:29:57.169634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.169606    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:29:57.267851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.267814    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.267866    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.267896    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.267958    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268023    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\" (UniqueName: \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268066    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268109    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268156    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268155    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.268315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268266    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268304    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\") pod \"35c8ac26-d502-4557-a7d6-535211e12e8b\" (UID: \"35c8ac26-d502-4557-a7d6-535211e12e8b\") "
Oct 30 22:29:57.268543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268333    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268374    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\" (UniqueName: \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\") pod \"405f22c9-78f4-4763-95d0-444646c01cf4\" (UID: \"405f22c9-78f4-4763-95d0-444646c01cf4\") "
Oct 30 22:29:57.268543 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268373    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.268727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268578    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.268946 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268891    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.268946 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268913    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.268946 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.268928    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.269448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.269281    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.269448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.269369    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.269448 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.269401    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.271141 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.271101    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:57.271253 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.271233    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "kube-api-access-69n9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:57.271446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.271419    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "kube-api-access-wrzsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:57.271615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.271587    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm" pod "35c8ac26-d502-4557-a7d6-535211e12e8b" (UID: "35c8ac26-d502-4557-a7d6-535211e12e8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.271709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.271620    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:57.272140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.272119    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm" pod "405f22c9-78f4-4763-95d0-444646c01cf4" (UID: "405f22c9-78f4-4763-95d0-444646c01cf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369625    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369651    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369677    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f22c9-78f4-4763-95d0-444646c01cf4-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369687    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrzsr\" (UniqueName: \"kubernetes.io/projected/35c8ac26-d502-4557-a7d6-535211e12e8b-kube-api-access-wrzsr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369697    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c8ac26-d502-4557-a7d6-535211e12e8b-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369705    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369714    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c8ac26-d502-4557-a7d6-535211e12e8b-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.369717 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369722    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f22c9-78f4-4763-95d0-444646c01cf4-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.370286 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.369730    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n9f\" (UniqueName: \"kubernetes.io/projected/405f22c9-78f4-4763-95d0-444646c01cf4-kube-api-access-69n9f\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.727170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.727131    4295 generic.go:356] "Generic (PLEG): container finished" podID="405f22c9-78f4-4763-95d0-444646c01cf4" containerID="45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7" exitCode=0
Oct 30 22:29:57.727377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.727184    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"
Oct 30 22:29:57.727377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.727223    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" event={"ID":"405f22c9-78f4-4763-95d0-444646c01cf4","Type":"ContainerDied","Data":"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"}
Oct 30 22:29:57.727377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.727276    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j" event={"ID":"405f22c9-78f4-4763-95d0-444646c01cf4","Type":"ContainerDied","Data":"c627133a0581cc6aeb1765391054974bd420828fee61fccf496734d6bc03d193"}
Oct 30 22:29:57.727377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.727300    4295 scope.go:122] "RemoveContainer" containerID="45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"
Oct 30 22:29:57.728604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.728576    4295 generic.go:356] "Generic (PLEG): container finished" podID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerID="ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac" exitCode=0
Oct 30 22:29:57.728742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.728610    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" event={"ID":"35c8ac26-d502-4557-a7d6-535211e12e8b","Type":"ContainerDied","Data":"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"}
Oct 30 22:29:57.728742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.728633    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"
Oct 30 22:29:57.728742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.728687    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm" event={"ID":"35c8ac26-d502-4557-a7d6-535211e12e8b","Type":"ContainerDied","Data":"b74ffadd05766253f5d5ffaee8c1932220cbc855b0934abdd2a936dd03e6c684"}
Oct 30 22:29:57.739560 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.739545    4295 scope.go:122] "RemoveContainer" containerID="45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"
Oct 30 22:29:57.739840 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:29:57.739825    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7\": container with ID starting with 45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7 not found: ID does not exist" containerID="45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"
Oct 30 22:29:57.739904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.739848    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7"} err="failed to get container status \"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7\": rpc error: code = NotFound desc = could not find container \"45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7\": container with ID starting with 45406e23d4f237b30d0d879396da436c390e343b47ac6308ddbc7429dc6adaf7 not found: ID does not exist"
Oct 30 22:29:57.739904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.739864    4295 scope.go:122] "RemoveContainer" containerID="ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"
Oct 30 22:29:57.747523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.747494    4295 scope.go:122] "RemoveContainer" containerID="ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"
Oct 30 22:29:57.747781 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:29:57.747761    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac\": container with ID starting with ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac not found: ID does not exist" containerID="ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"
Oct 30 22:29:57.747851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.747789    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac"} err="failed to get container status \"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac\": rpc error: code = NotFound desc = could not find container \"ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac\": container with ID starting with ec91462923adea587c3dc1cef8ea3684d28cfbe8bb70c35e5ee2342aac6942ac not found: ID does not exist"
Oct 30 22:29:57.898865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.898838    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:29:57.948495 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:57.948465    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-w6-2182a5-kserve-97c95cdb6-mjhcm"]
Oct 30 22:29:58.033726 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:58.033611    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:29:58.041446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:58.041416    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/distinct-2-w6-2182a5-kserve-68b654cb97-7qb5j"]
Oct 30 22:29:58.997357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:58.997305    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:58.997357 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:58.997339    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:59.493013 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:59.492980    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" path="/var/lib/kubelet/pods/35c8ac26-d502-4557-a7d6-535211e12e8b/volumes"
Oct 30 22:29:59.493366 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:29:59.493353    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" path="/var/lib/kubelet/pods/405f22c9-78f4-4763-95d0-444646c01cf4/volumes"
Oct 30 22:30:00.166385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166346    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:00.166791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166776    4295 cpu_manager.go:414] "containerMap: removing container" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166792    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166806    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166812    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166825    4295 cpu_manager.go:414] "containerMap: removing container" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166830    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166840    4295 cpu_manager.go:414] "containerMap: removing container" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main"
Oct 30 22:30:00.166852 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166844    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166859    4295 cpu_manager.go:414] "containerMap: removing container" podUID="5b823cba-638a-4058-9635-fa8604d96c87" containerName="revoke-keys"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166864    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b823cba-638a-4058-9635-fa8604d96c87" containerName="revoke-keys"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166920    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f22c9-78f4-4763-95d0-444646c01cf4" containerName="main"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166928    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c8ac26-d502-4557-a7d6-535211e12e8b" containerName="main"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166937    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b823cba-638a-4058-9635-fa8604d96c87" containerName="revoke-keys"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166941    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd26d361-a47d-4531-a0e5-74ee9b9b7a05" containerName="istio-proxy"
Oct 30 22:30:00.167106 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.166950    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9350b3-2120-461c-9eb3-4144ed6f0a2f" containerName="istio-proxy"
Oct 30 22:30:00.171600 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.171579    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:00.171732 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.171715    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.174968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.174951    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.228219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.228183    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:30:00.287850 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.287816    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:00.291230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.291204    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:00.300822 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.300793    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.300981 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.300961    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xfc\" (UniqueName: \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.301062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.301011    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54zq\" (UniqueName: \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.301138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.301113    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.403768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.402769    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xfc\" (UniqueName: \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.403768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.402839    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54zq\" (UniqueName: \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.403768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.402875    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.403768 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.402999    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.404140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.403996    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.404610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.404581    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.467345 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.467313    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54zq\" (UniqueName: \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.467646 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.467625    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xfc\" (UniqueName: \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.485725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.485703    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:00.492560 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.492539    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:00.863648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.863620    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:00.865015 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:00.864984    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef297b82_0550_4038_bb69_fcf766aab2d5.slice/crio-e9ce7c1b13be108c8047bfd495e969d4fc0ea278bb1ff1052d91cd8d14f90c1a WatchSource:0}: Error finding container e9ce7c1b13be108c8047bfd495e969d4fc0ea278bb1ff1052d91cd8d14f90c1a: Status 404 returned error can't find the container with id e9ce7c1b13be108c8047bfd495e969d4fc0ea278bb1ff1052d91cd8d14f90c1a
Oct 30 22:30:00.912737 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:00.912711    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:00.914837 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:00.914806    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5f1641_9264_4643_a750_a497ba1254da.slice/crio-a81a0f3fb7578a411e67269c654d7f1cc8d0cebda2937d5255b43210a25b8581 WatchSource:0}: Error finding container a81a0f3fb7578a411e67269c654d7f1cc8d0cebda2937d5255b43210a25b8581: Status 404 returned error can't find the container with id a81a0f3fb7578a411e67269c654d7f1cc8d0cebda2937d5255b43210a25b8581
Oct 30 22:30:01.749373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.749334    4295 generic.go:356] "Generic (PLEG): container finished" podID="2a5f1641-9264-4643-a750-a497ba1254da" containerID="aa5e07782d2b97c62859af1ec97ba8d83abcf9c01c6165e8306e86b576d47cb6" exitCode=60
Oct 30 22:30:01.749869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.749423    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerDied","Data":"aa5e07782d2b97c62859af1ec97ba8d83abcf9c01c6165e8306e86b576d47cb6"}
Oct 30 22:30:01.749869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.749478    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerStarted","Data":"a81a0f3fb7578a411e67269c654d7f1cc8d0cebda2937d5255b43210a25b8581"}
Oct 30 22:30:01.749869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.749691    4295 scope.go:122] "RemoveContainer" containerID="aa5e07782d2b97c62859af1ec97ba8d83abcf9c01c6165e8306e86b576d47cb6"
Oct 30 22:30:01.751187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.751071    4295 generic.go:356] "Generic (PLEG): container finished" podID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerID="9818e5a76f3728df59580aab374c0798b21cdc473fc683dd683507eb05a9f11f" exitCode=60
Oct 30 22:30:01.751187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.751112    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerDied","Data":"9818e5a76f3728df59580aab374c0798b21cdc473fc683dd683507eb05a9f11f"}
Oct 30 22:30:01.751187 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.751169    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerStarted","Data":"e9ce7c1b13be108c8047bfd495e969d4fc0ea278bb1ff1052d91cd8d14f90c1a"}
Oct 30 22:30:01.751419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:01.751403    4295 scope.go:122] "RemoveContainer" containerID="9818e5a76f3728df59580aab374c0798b21cdc473fc683dd683507eb05a9f11f"
Oct 30 22:30:02.759610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.759513    4295 generic.go:356] "Generic (PLEG): container finished" podID="2a5f1641-9264-4643-a750-a497ba1254da" containerID="14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c" exitCode=60
Oct 30 22:30:02.760105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.759646    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerDied","Data":"14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c"}
Oct 30 22:30:02.760105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.759744    4295 scope.go:122] "RemoveContainer" containerID="aa5e07782d2b97c62859af1ec97ba8d83abcf9c01c6165e8306e86b576d47cb6"
Oct 30 22:30:02.760105 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.760014    4295 scope.go:122] "RemoveContainer" containerID="14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c"
Oct 30 22:30:02.760343 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:02.760321    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd_odh-ai-gateway-infra(2a5f1641-9264-4643-a750-a497ba1254da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" podUID="2a5f1641-9264-4643-a750-a497ba1254da"
Oct 30 22:30:02.762242 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.762220    4295 generic.go:356] "Generic (PLEG): container finished" podID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerID="e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39" exitCode=60
Oct 30 22:30:02.762347 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.762250    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerDied","Data":"e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39"}
Oct 30 22:30:02.762559 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.762543    4295 scope.go:122] "RemoveContainer" containerID="e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39"
Oct 30 22:30:02.762869 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:02.762843    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj_odh-ai-gateway-infra(ef297b82-0550-4038-bb69-fcf766aab2d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5"
Oct 30 22:30:02.772038 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:02.772004    4295 scope.go:122] "RemoveContainer" containerID="9818e5a76f3728df59580aab374c0798b21cdc473fc683dd683507eb05a9f11f"
Oct 30 22:30:03.805702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:03.803877    4295 scope.go:122] "RemoveContainer" containerID="14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c"
Oct 30 22:30:03.805702 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:03.804334    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd_odh-ai-gateway-infra(2a5f1641-9264-4643-a750-a497ba1254da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" podUID="2a5f1641-9264-4643-a750-a497ba1254da"
Oct 30 22:30:03.830701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:03.830573    4295 scope.go:122] "RemoveContainer" containerID="e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39"
Oct 30 22:30:03.834695 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:03.830947    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj_odh-ai-gateway-infra(ef297b82-0550-4038-bb69-fcf766aab2d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5"
Oct 30 22:30:05.037693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.025526    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"]
Oct 30 22:30:05.048074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.048043    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.085190 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.085147    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"]
Oct 30 22:30:05.109687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.104965    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:05.159340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.152836    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.159340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.158554    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.159340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.158615    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qq4\" (UniqueName: \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.259747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.259710    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.260007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.259775    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.260007 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.259809    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qq4\" (UniqueName: \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.260400 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.260377    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.262364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.262335    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.322995 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.322924    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qq4\" (UniqueName: \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\") pod \"maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.375649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.375625    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:05.750934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.750905    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"]
Oct 30 22:30:05.752442 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:05.752418    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80e475b_061a_4880_ad2e_949b7bd29297.slice/crio-246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c WatchSource:0}: Error finding container 246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c: Status 404 returned error can't find the container with id 246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c
Oct 30 22:30:05.853138 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:05.853107    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d" event={"ID":"b80e475b-061a-4880-ad2e-949b7bd29297","Type":"ContainerStarted","Data":"246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c"}
Oct 30 22:30:06.859673 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:06.859625    4295 generic.go:356] "Generic (PLEG): container finished" podID="b80e475b-061a-4880-ad2e-949b7bd29297" containerID="a3975f1f420bdeb6ad14acb28356534208745bc43e4d030a0e06833b2fbb2d5e" exitCode=0
Oct 30 22:30:06.860092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:06.859708    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d" event={"ID":"b80e475b-061a-4880-ad2e-949b7bd29297","Type":"ContainerDied","Data":"a3975f1f420bdeb6ad14acb28356534208745bc43e4d030a0e06833b2fbb2d5e"}
Oct 30 22:30:08.003836 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.003811    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:08.099720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.099678    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\" (UniqueName: \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\") pod \"b80e475b-061a-4880-ad2e-949b7bd29297\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") "
Oct 30 22:30:08.099884 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.099724    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\") pod \"b80e475b-061a-4880-ad2e-949b7bd29297\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") "
Oct 30 22:30:08.099884 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.099800    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\") pod \"b80e475b-061a-4880-ad2e-949b7bd29297\" (UID: \"b80e475b-061a-4880-ad2e-949b7bd29297\") "
Oct 30 22:30:08.100210 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.100180    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca" pod "b80e475b-061a-4880-ad2e-949b7bd29297" (UID: "b80e475b-061a-4880-ad2e-949b7bd29297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:08.101950 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.101928    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca" pod "b80e475b-061a-4880-ad2e-949b7bd29297" (UID: "b80e475b-061a-4880-ad2e-949b7bd29297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:08.102030 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.102015    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4" pod "b80e475b-061a-4880-ad2e-949b7bd29297" (UID: "b80e475b-061a-4880-ad2e-949b7bd29297"). InnerVolumeSpecName "kube-api-access-47qq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:08.201331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.201298    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.201331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.201326    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qq4\" (UniqueName: \"kubernetes.io/projected/b80e475b-061a-4880-ad2e-949b7bd29297-kube-api-access-47qq4\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.201331 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.201338    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e475b-061a-4880-ad2e-949b7bd29297-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:08.871730 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.871704    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"
Oct 30 22:30:08.871908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.871694    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d" event={"ID":"b80e475b-061a-4880-ad2e-949b7bd29297","Type":"ContainerDied","Data":"246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c"}
Oct 30 22:30:08.871908 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.871834    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246324496a58314d8e049e95e0ecb006b786332dfaa199ba447edfab281bb38c"
Oct 30 22:30:08.996533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.996493    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:08.996709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:08.996493    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:10.705177 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.705142    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"]
Oct 30 22:30:10.705631 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.705554    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b80e475b-061a-4880-ad2e-949b7bd29297" containerName="revoke-keys"
Oct 30 22:30:10.705631 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.705564    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80e475b-061a-4880-ad2e-949b7bd29297" containerName="revoke-keys"
Oct 30 22:30:10.705727 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.705654    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80e475b-061a-4880-ad2e-949b7bd29297" containerName="revoke-keys"
Oct 30 22:30:10.708770 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.708754    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.768696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.768641    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"]
Oct 30 22:30:10.814230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.814153    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:10.830575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.830543    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfcp\" (UniqueName: \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.830753 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.830729    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.830796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.830780    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.931925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.931891    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.932114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.931933    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.932114 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.932025    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfcp\" (UniqueName: \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.932672 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.932641    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.934315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.934293    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:10.995654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:10.995593    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfcp\" (UniqueName: \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\") pod \"maas-api-revoke-sub-32203fc5-97gp9\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:11.023517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.023492    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:11.330255 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.330153    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main" containerID="cri-o://544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4" gracePeriod=45
Oct 30 22:30:11.379716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.379646    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main" containerID="cri-o://3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358" gracePeriod=45
Oct 30 22:30:11.398379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.398358    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"]
Oct 30 22:30:11.442547 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:11.442516    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd8804d_5d51_4a05_94d5_e3642d47b1a0.slice/crio-7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc WatchSource:0}: Error finding container 7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc: Status 404 returned error can't find the container with id 7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc
Oct 30 22:30:11.668592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.668563    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:30:11.694799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.694775    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:30:11.741304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741268    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741327    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\" (UniqueName: \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741374    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741406    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741446    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741488    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\" (UniqueName: \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741522    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\" (UniqueName: \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741565    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.741762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741656    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741837    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.742198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741878    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741941    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.742198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.741989    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\") pod \"05ec1674-4204-4c78-8c0e-fc4882173d37\" (UID: \"05ec1674-4204-4c78-8c0e-fc4882173d37\") "
Oct 30 22:30:11.742198 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742025    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\") pod \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\" (UID: \"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d\") "
Oct 30 22:30:11.742466 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742282    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742466 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742310    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742477    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742486    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.742571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742562    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.742762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742583    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.742762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742599    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.742762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742613    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.742762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.742626    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.744599 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.744570    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:11.745524 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.745502    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.746398 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.746294    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "kube-api-access-blmxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:11.746398 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.746330    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.746508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.746397    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf" pod "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" (UID: "8f23d29b-ea29-4dc6-8d97-a7b4bf85083d"). InnerVolumeSpecName "kube-api-access-c6cxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:11.746639 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.746609    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs" pod "05ec1674-4204-4c78-8c0e-fc4882173d37" (UID: "05ec1674-4204-4c78-8c0e-fc4882173d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:11.843336 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843307    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843336 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843334    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843344    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmxx\" (UniqueName: \"kubernetes.io/projected/05ec1674-4204-4c78-8c0e-fc4882173d37-kube-api-access-blmxx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843352    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ec1674-4204-4c78-8c0e-fc4882173d37-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843361    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ec1674-4204-4c78-8c0e-fc4882173d37-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843369    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6cxf\" (UniqueName: \"kubernetes.io/projected/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-kube-api-access-c6cxf\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.843522 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.843377    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.889866 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.889826    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9" event={"ID":"cdd8804d-5d51-4a05-94d5-e3642d47b1a0","Type":"ContainerStarted","Data":"7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc"}
Oct 30 22:30:11.891449 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.891423    4295 generic.go:356] "Generic (PLEG): container finished" podID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerID="544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4" exitCode=0
Oct 30 22:30:11.891591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.891476    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"
Oct 30 22:30:11.891591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.891500    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" event={"ID":"05ec1674-4204-4c78-8c0e-fc4882173d37","Type":"ContainerDied","Data":"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"}
Oct 30 22:30:11.891591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.891534    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj" event={"ID":"05ec1674-4204-4c78-8c0e-fc4882173d37","Type":"ContainerDied","Data":"e2fe5bbe7885047ecfa095f518f87119f442c1407c03f6a9d4abbad83e7e8a80"}
Oct 30 22:30:11.891591 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.891557    4295 scope.go:122] "RemoveContainer" containerID="544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"
Oct 30 22:30:11.893093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.892990    4295 generic.go:356] "Generic (PLEG): container finished" podID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerID="3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358" exitCode=0
Oct 30 22:30:11.893093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.893039    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"
Oct 30 22:30:11.893093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.893069    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" event={"ID":"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d","Type":"ContainerDied","Data":"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"}
Oct 30 22:30:11.893300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.893106    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct" event={"ID":"8f23d29b-ea29-4dc6-8d97-a7b4bf85083d","Type":"ContainerDied","Data":"5502d4fd7f662ba3ee9978852a5b15514a957873405f90cfe9298a1239f2315f"}
Oct 30 22:30:11.906440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.904203    4295 scope.go:122] "RemoveContainer" containerID="544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"
Oct 30 22:30:11.906440 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:11.904621    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4\": container with ID starting with 544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4 not found: ID does not exist" containerID="544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"
Oct 30 22:30:11.906440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.904655    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4"} err="failed to get container status \"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4\": rpc error: code = NotFound desc = could not find container \"544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4\": container with ID starting with 544f7aa2e75fd602cf66ba392ec3d11945b1072ba729dfba4886f6fffc3f7bf4 not found: ID does not exist"
Oct 30 22:30:11.906440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.904697    4295 scope.go:122] "RemoveContainer" containerID="3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"
Oct 30 22:30:11.919494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.919471    4295 scope.go:122] "RemoveContainer" containerID="3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"
Oct 30 22:30:11.919836 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:11.919816    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358\": container with ID starting with 3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358 not found: ID does not exist" containerID="3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"
Oct 30 22:30:11.919927 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:11.919844    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358"} err="failed to get container status \"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358\": rpc error: code = NotFound desc = could not find container \"3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358\": container with ID starting with 3a12497c7d49c90087ce8810ae07a7f1948b780d0031945d332ea964125c2358 not found: ID does not exist"
Oct 30 22:30:12.133515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.133483    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:30:12.143701 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.143643    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/simulator-w5-3783c2-kserve-fbc59d5db-b4jct"]
Oct 30 22:30:12.251693 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.251648    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:30:12.261414 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.261389    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-2-w5-3783c2-kserve-5f6bf8646f-s94jj"]
Oct 30 22:30:12.901848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.901816    4295 generic.go:356] "Generic (PLEG): container finished" podID="cdd8804d-5d51-4a05-94d5-e3642d47b1a0" containerID="a77adfc3851ca9d2d1f46ed2af99d51dc80af8504dac95fcd00689371fb8d885" exitCode=0
Oct 30 22:30:12.902206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:12.901856    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9" event={"ID":"cdd8804d-5d51-4a05-94d5-e3642d47b1a0","Type":"ContainerDied","Data":"a77adfc3851ca9d2d1f46ed2af99d51dc80af8504dac95fcd00689371fb8d885"}
Oct 30 22:30:13.088971 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.088947    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:30:13.155967 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.155939    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\") pod \"0bd3562e-550c-4567-be9a-cdad66c4d393\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") "
Oct 30 22:30:13.156142 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.156076    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\" (UniqueName: \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\") pod \"0bd3562e-550c-4567-be9a-cdad66c4d393\" (UID: \"0bd3562e-550c-4567-be9a-cdad66c4d393\") "
Oct 30 22:30:13.156418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.156390    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume" pod "0bd3562e-550c-4567-be9a-cdad66c4d393" (UID: "0bd3562e-550c-4567-be9a-cdad66c4d393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:13.158447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.158420    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj" pod "0bd3562e-550c-4567-be9a-cdad66c4d393" (UID: "0bd3562e-550c-4567-be9a-cdad66c4d393"). InnerVolumeSpecName "kube-api-access-fbprj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:13.214772 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.214745    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:30:13.257613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.257573    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbprj\" (UniqueName: \"kubernetes.io/projected/0bd3562e-550c-4567-be9a-cdad66c4d393-kube-api-access-fbprj\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.257613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.257613    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3562e-550c-4567-be9a-cdad66c4d393-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.358309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.358279    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\") pod \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") "
Oct 30 22:30:13.358479 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.358420    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\" (UniqueName: \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\") pod \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\" (UID: \"637c3951-da90-46f4-b6a3-092b3d4cc3c4\") "
Oct 30 22:30:13.358684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.358639    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume" pod "637c3951-da90-46f4-b6a3-092b3d4cc3c4" (UID: "637c3951-da90-46f4-b6a3-092b3d4cc3c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:13.358837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.358814    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/637c3951-da90-46f4-b6a3-092b3d4cc3c4-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.360725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.360698    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z" pod "637c3951-da90-46f4-b6a3-092b3d4cc3c4" (UID: "637c3951-da90-46f4-b6a3-092b3d4cc3c4"). InnerVolumeSpecName "kube-api-access-5c26z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:13.459803 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.459762    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c26z\" (UniqueName: \"kubernetes.io/projected/637c3951-da90-46f4-b6a3-092b3d4cc3c4-kube-api-access-5c26z\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.487122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.487093    4295 scope.go:122] "RemoveContainer" containerID="e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39"
Oct 30 22:30:13.493758 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.493696    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" path="/var/lib/kubelet/pods/05ec1674-4204-4c78-8c0e-fc4882173d37/volumes"
Oct 30 22:30:13.494098 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.494084    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" path="/var/lib/kubelet/pods/8f23d29b-ea29-4dc6-8d97-a7b4bf85083d/volumes"
Oct 30 22:30:13.910055 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.910020    4295 generic.go:356] "Generic (PLEG): container finished" podID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerID="e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400" exitCode=0
Oct 30 22:30:13.910493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.910079    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"
Oct 30 22:30:13.910493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.910095    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" event={"ID":"0bd3562e-550c-4567-be9a-cdad66c4d393","Type":"ContainerDied","Data":"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"}
Oct 30 22:30:13.910493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.910166    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv" event={"ID":"0bd3562e-550c-4567-be9a-cdad66c4d393","Type":"ContainerDied","Data":"ca6586c8b3fae33c0a31b5e1ebbbad770073e56ad44c2d413faf4ecc0fe26179"}
Oct 30 22:30:13.910493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.910194    4295 scope.go:122] "RemoveContainer" containerID="e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"
Oct 30 22:30:13.912239 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.912139    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerStarted","Data":"c28914397b692e8e2e7875f3f1a53f60295646d36448717dc7e2356b0bace0cf"}
Oct 30 22:30:13.913718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.913697    4295 generic.go:356] "Generic (PLEG): container finished" podID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerID="a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c" exitCode=0
Oct 30 22:30:13.913815 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.913791    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" event={"ID":"637c3951-da90-46f4-b6a3-092b3d4cc3c4","Type":"ContainerDied","Data":"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"}
Oct 30 22:30:13.913871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.913834    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f" event={"ID":"637c3951-da90-46f4-b6a3-092b3d4cc3c4","Type":"ContainerDied","Data":"bd1bc98e8f1f28302863592104f097a668b8d66444f273a57be639cd11212a20"}
Oct 30 22:30:13.913871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.913847    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"
Oct 30 22:30:13.920074 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.920044    4295 scope.go:122] "RemoveContainer" containerID="e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"
Oct 30 22:30:13.920360 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:13.920341    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400\": container with ID starting with e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400 not found: ID does not exist" containerID="e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"
Oct 30 22:30:13.920413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.920367    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400"} err="failed to get container status \"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400\": rpc error: code = NotFound desc = could not find container \"e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400\": container with ID starting with e57ffbf756d00c97ca22da8586351abd1f6740e1675bfc0e709c60dfbfe50400 not found: ID does not exist"
Oct 30 22:30:13.920413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.920385    4295 scope.go:122] "RemoveContainer" containerID="a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"
Oct 30 22:30:13.928817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.928794    4295 scope.go:122] "RemoveContainer" containerID="a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"
Oct 30 22:30:13.929119 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:13.929093    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c\": container with ID starting with a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c not found: ID does not exist" containerID="a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"
Oct 30 22:30:13.929185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:13.929129    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c"} err="failed to get container status \"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c\": rpc error: code = NotFound desc = could not find container \"a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c\": container with ID starting with a11cfa1464622e2dc64725c2d3d5c1356b5a1aa0ab417cdf36ce61112c6ca39c not found: ID does not exist"
Oct 30 22:30:14.057784 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.057729    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:14.082911 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.082882    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:30:14.085741 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.085716    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f"]
Oct 30 22:30:14.167556 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.167528    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\") pod \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") "
Oct 30 22:30:14.167729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.167588    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\") pod \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") "
Oct 30 22:30:14.167855 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.167827    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\" (UniqueName: \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\") pod \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\" (UID: \"cdd8804d-5d51-4a05-94d5-e3642d47b1a0\") "
Oct 30 22:30:14.167991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.167972    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca" pod "cdd8804d-5d51-4a05-94d5-e3642d47b1a0" (UID: "cdd8804d-5d51-4a05-94d5-e3642d47b1a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.168157 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.168141    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:14.169890 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.169866    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca" pod "cdd8804d-5d51-4a05-94d5-e3642d47b1a0" (UID: "cdd8804d-5d51-4a05-94d5-e3642d47b1a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:14.170153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.170131    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp" pod "cdd8804d-5d51-4a05-94d5-e3642d47b1a0" (UID: "cdd8804d-5d51-4a05-94d5-e3642d47b1a0"). InnerVolumeSpecName "kube-api-access-ptfcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.198613 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.198572    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:30:14.204493 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.204471    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv"]
Oct 30 22:30:14.268749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.268717    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptfcp\" (UniqueName: \"kubernetes.io/projected/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-kube-api-access-ptfcp\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:14.268749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.268744    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd8804d-5d51-4a05-94d5-e3642d47b1a0-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:14.322612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.322512    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" podStartSLOduration=14.322496567 podStartE2EDuration="14.322496567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:14.320862634 +0000 UTC m=+2416.963456423" watchObservedRunningTime="2026-10-30 22:30:14.322496567 +0000 UTC m=+2416.965090355"
Oct 30 22:30:14.486851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.486818    4295 scope.go:122] "RemoveContainer" containerID="14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c"
Oct 30 22:30:14.921720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.921682    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerStarted","Data":"e878842d6e582a295588a1b491fbc6f89756bf324aab585eb359231eab81dc59"}
Oct 30 22:30:14.923581 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.923555    4295 generic.go:356] "Generic (PLEG): container finished" podID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerID="c28914397b692e8e2e7875f3f1a53f60295646d36448717dc7e2356b0bace0cf" exitCode=60
Oct 30 22:30:14.923733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.923628    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerDied","Data":"c28914397b692e8e2e7875f3f1a53f60295646d36448717dc7e2356b0bace0cf"}
Oct 30 22:30:14.923733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.923703    4295 scope.go:122] "RemoveContainer" containerID="e97d24142a11d60c8ae9e2df81786a20b817fcb063cf80c89984656b3e7c9c39"
Oct 30 22:30:14.924044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.924029    4295 scope.go:122] "RemoveContainer" containerID="c28914397b692e8e2e7875f3f1a53f60295646d36448717dc7e2356b0bace0cf"
Oct 30 22:30:14.924350 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:14.924322    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj_odh-ai-gateway-infra(ef297b82-0550-4038-bb69-fcf766aab2d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5"
Oct 30 22:30:14.927084 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.927061    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9" event={"ID":"cdd8804d-5d51-4a05-94d5-e3642d47b1a0","Type":"ContainerDied","Data":"7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc"}
Oct 30 22:30:14.927175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.927098    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"
Oct 30 22:30:14.927175 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:14.927105    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7adce6c055c67e25ce9b4b0215228c220481bfad317eb8521c91c5be09e7b4bc"
Oct 30 22:30:15.043188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.043127    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" podStartSLOduration=15.043111688 podStartE2EDuration="15.043111688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:15.041920648 +0000 UTC m=+2417.684514450" watchObservedRunningTime="2026-10-30 22:30:15.043111688 +0000 UTC m=+2417.685705477"
Oct 30 22:30:15.332894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.332861    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:15.492011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.491978    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" path="/var/lib/kubelet/pods/0bd3562e-550c-4567-be9a-cdad66c4d393/volumes"
Oct 30 22:30:15.492364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.492351    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" path="/var/lib/kubelet/pods/637c3951-da90-46f4-b6a3-092b3d4cc3c4/volumes"
Oct 30 22:30:16.003170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.986717    4295 generic.go:356] "Generic (PLEG): container finished" podID="2a5f1641-9264-4643-a750-a497ba1254da" containerID="e878842d6e582a295588a1b491fbc6f89756bf324aab585eb359231eab81dc59" exitCode=60
Oct 30 22:30:16.003170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.986790    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerDied","Data":"e878842d6e582a295588a1b491fbc6f89756bf324aab585eb359231eab81dc59"}
Oct 30 22:30:16.003170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.986868    4295 scope.go:122] "RemoveContainer" containerID="14dcec7bd16dc47d1a54e240511f032ac210c4faf6e7475feedcb14a0ffffc7c"
Oct 30 22:30:16.003170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:15.990721    4295 scope.go:122] "RemoveContainer" containerID="e878842d6e582a295588a1b491fbc6f89756bf324aab585eb359231eab81dc59"
Oct 30 22:30:16.003170 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:15.991446    4295 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd_odh-ai-gateway-infra(2a5f1641-9264-4643-a750-a497ba1254da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" podUID="2a5f1641-9264-4643-a750-a497ba1254da"
Oct 30 22:30:16.056428 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.056387    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:16.179643 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.179618    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"]
Oct 30 22:30:16.180095 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180081    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main"
Oct 30 22:30:16.180170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180099    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main"
Oct 30 22:30:16.180170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180117    4295 cpu_manager.go:414] "containerMap: removing container" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main"
Oct 30 22:30:16.180170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180121    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main"
Oct 30 22:30:16.180170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180157    4295 cpu_manager.go:414] "containerMap: removing container" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing"
Oct 30 22:30:16.180170 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180171    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180182    4295 cpu_manager.go:414] "containerMap: removing container" podUID="cdd8804d-5d51-4a05-94d5-e3642d47b1a0" containerName="revoke-keys"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180187    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd8804d-5d51-4a05-94d5-e3642d47b1a0" containerName="revoke-keys"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180194    4295 cpu_manager.go:414] "containerMap: removing container" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180198    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180264    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd3562e-550c-4567-be9a-cdad66c4d393" containerName="payload-processing"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180273    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd8804d-5d51-4a05-94d5-e3642d47b1a0" containerName="revoke-keys"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180279    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="637c3951-da90-46f4-b6a3-092b3d4cc3c4" containerName="payload-pre-processing"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180286    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f23d29b-ea29-4dc6-8d97-a7b4bf85083d" containerName="main"
Oct 30 22:30:16.180330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.180293    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ec1674-4204-4c78-8c0e-fc4882173d37" containerName="main"
Oct 30 22:30:16.183898 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.183865    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:16.184675 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.184640    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.239699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.239602    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:16.240777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.240751    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"]
Oct 30 22:30:16.254403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254376    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"]
Oct 30 22:30:16.254790 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254777    4295 cpu_manager.go:414] "containerMap: removing container" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.254861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254791    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.254861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254818    4295 cpu_manager.go:414] "containerMap: removing container" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.254861 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254823    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.255011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254889    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.255011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.254907    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:16.258116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.258100    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.295766 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.295731    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\" (UniqueName: \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\") pod \"ef297b82-0550-4038-bb69-fcf766aab2d5\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") "
Oct 30 22:30:16.295914 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.295835    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\") pod \"ef297b82-0550-4038-bb69-fcf766aab2d5\" (UID: \"ef297b82-0550-4038-bb69-fcf766aab2d5\") "
Oct 30 22:30:16.296076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.296058    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.296211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.296184    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca" pod "ef297b82-0550-4038-bb69-fcf766aab2d5" (UID: "ef297b82-0550-4038-bb69-fcf766aab2d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.296279 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.296218    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.296333 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.296281    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnph\" (UniqueName: \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.296443 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.296420    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef297b82-0550-4038-bb69-fcf766aab2d5-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.298098 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.298070    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc" pod "ef297b82-0550-4038-bb69-fcf766aab2d5" (UID: "ef297b82-0550-4038-bb69-fcf766aab2d5"). InnerVolumeSpecName "kube-api-access-v4xfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.321554 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.321526    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"]
Oct 30 22:30:16.397618 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397586    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwctj\" (UniqueName: \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.397811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397628    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.397811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397769    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnph\" (UniqueName: \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.397889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397812    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.397889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397868    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.397949 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.397933    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.398057 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.398026    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xfc\" (UniqueName: \"kubernetes.io/projected/ef297b82-0550-4038-bb69-fcf766aab2d5-kube-api-access-v4xfc\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.398462 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.398443    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.400282 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.400257    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.462742 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.462705    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnph\" (UniqueName: \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\") pod \"maas-api-revoke-sub-9c74b376-qq644\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.498185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.498109    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:16.499562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.499535    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.499759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.499603    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.499759 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.499656    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwctj\" (UniqueName: \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.500750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.500720    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.502181 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.502100    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.563051 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.563019    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwctj\" (UniqueName: \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\") pod \"maas-api-revoke-sub-9a30b195-ggfsj\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.572791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.572765    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:16.884293 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:16.884195    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dece7e_3005_418e_9c02_3dc0bf5daf6e.slice/crio-0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db WatchSource:0}: Error finding container 0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db: Status 404 returned error can't find the container with id 0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db
Oct 30 22:30:16.886975 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.886489    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"]
Oct 30 22:30:16.957899 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.957869    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"]
Oct 30 22:30:16.959141 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:16.959116    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb540a89a_fbb7_4439_b07d_cbfa67ce7c8a.slice/crio-3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f WatchSource:0}: Error finding container 3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f: Status 404 returned error can't find the container with id 3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f
Oct 30 22:30:16.994945 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.994925    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"
Oct 30 22:30:16.995046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.994925    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj" event={"ID":"ef297b82-0550-4038-bb69-fcf766aab2d5","Type":"ContainerDied","Data":"e9ce7c1b13be108c8047bfd495e969d4fc0ea278bb1ff1052d91cd8d14f90c1a"}
Oct 30 22:30:16.995093 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.995076    4295 scope.go:122] "RemoveContainer" containerID="c28914397b692e8e2e7875f3f1a53f60295646d36448717dc7e2356b0bace0cf"
Oct 30 22:30:16.996220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.996199    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644" event={"ID":"c7dece7e-3005-418e-9c02-3dc0bf5daf6e","Type":"ContainerStarted","Data":"0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db"}
Oct 30 22:30:16.997597 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:16.997562    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj" event={"ID":"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a","Type":"ContainerStarted","Data":"3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f"}
Oct 30 22:30:17.226880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.226847    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:17.253796 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.253763    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:17.256422 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.256393    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3783c2-29889990-htcxj"]
Oct 30 22:30:17.313271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.313241    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\") pod \"2a5f1641-9264-4643-a750-a497ba1254da\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") "
Oct 30 22:30:17.313401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.313280    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\" (UniqueName: \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\") pod \"2a5f1641-9264-4643-a750-a497ba1254da\" (UID: \"2a5f1641-9264-4643-a750-a497ba1254da\") "
Oct 30 22:30:17.313628 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.313605    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca" pod "2a5f1641-9264-4643-a750-a497ba1254da" (UID: "2a5f1641-9264-4643-a750-a497ba1254da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.315582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.315553    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq" pod "2a5f1641-9264-4643-a750-a497ba1254da" (UID: "2a5f1641-9264-4643-a750-a497ba1254da"). InnerVolumeSpecName "kube-api-access-n54zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.415343 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.415311    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f1641-9264-4643-a750-a497ba1254da-maas-api-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.415343 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.415340    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54zq\" (UniqueName: \"kubernetes.io/projected/2a5f1641-9264-4643-a750-a497ba1254da-kube-api-access-n54zq\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.493056 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:17.492984    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" path="/var/lib/kubelet/pods/ef297b82-0550-4038-bb69-fcf766aab2d5/volumes"
Oct 30 22:30:18.003647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.003620    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"
Oct 30 22:30:18.003647 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.003637    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd" event={"ID":"2a5f1641-9264-4643-a750-a497ba1254da","Type":"ContainerDied","Data":"a81a0f3fb7578a411e67269c654d7f1cc8d0cebda2937d5255b43210a25b8581"}
Oct 30 22:30:18.003885 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.003711    4295 scope.go:122] "RemoveContainer" containerID="e878842d6e582a295588a1b491fbc6f89756bf324aab585eb359231eab81dc59"
Oct 30 22:30:18.006818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.006795    4295 generic.go:356] "Generic (PLEG): container finished" podID="c7dece7e-3005-418e-9c02-3dc0bf5daf6e" containerID="ec7c08a25c29c5a9ceb7213353ac820acf972a6be382e07bff63d6e838e177b9" exitCode=0
Oct 30 22:30:18.006942 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.006878    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644" event={"ID":"c7dece7e-3005-418e-9c02-3dc0bf5daf6e","Type":"ContainerDied","Data":"ec7c08a25c29c5a9ceb7213353ac820acf972a6be382e07bff63d6e838e177b9"}
Oct 30 22:30:18.008587 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.008565    4295 generic.go:356] "Generic (PLEG): container finished" podID="b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" containerID="22dd10b894b4eb64c86afe62853d62595c6babd367fae81f05c8a6a4eb8b7df1" exitCode=0
Oct 30 22:30:18.008694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.008644    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj" event={"ID":"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a","Type":"ContainerDied","Data":"22dd10b894b4eb64c86afe62853d62595c6babd367fae81f05c8a6a4eb8b7df1"}
Oct 30 22:30:18.173378 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.173297    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:18.179506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:18.179480    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889990-b94dd"]
Oct 30 22:30:19.171235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.171213    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:19.179001 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.178980    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:19.233052 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233021    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\") pod \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") "
Oct 30 22:30:19.233265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233071    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\") pod \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") "
Oct 30 22:30:19.233265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233127    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\") pod \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") "
Oct 30 22:30:19.233265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233148    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\" (UniqueName: \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\") pod \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\" (UID: \"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a\") "
Oct 30 22:30:19.233265 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233212    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\") pod \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") "
Oct 30 22:30:19.233500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233275    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\" (UniqueName: \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\") pod \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\" (UID: \"c7dece7e-3005-418e-9c02-3dc0bf5daf6e\") "
Oct 30 22:30:19.233582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233535    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca" pod "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" (UID: "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.233777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233750    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.233777 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.233751    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca" pod "c7dece7e-3005-418e-9c02-3dc0bf5daf6e" (UID: "c7dece7e-3005-418e-9c02-3dc0bf5daf6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.235642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.235617    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca" pod "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" (UID: "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:19.236020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.235987    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca" pod "c7dece7e-3005-418e-9c02-3dc0bf5daf6e" (UID: "c7dece7e-3005-418e-9c02-3dc0bf5daf6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:19.236134 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.236108    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph" pod "c7dece7e-3005-418e-9c02-3dc0bf5daf6e" (UID: "c7dece7e-3005-418e-9c02-3dc0bf5daf6e"). InnerVolumeSpecName "kube-api-access-xfnph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.236202 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.236129    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj" pod "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" (UID: "b540a89a-fbb7-4439-b07d-cbfa67ce7c8a"). InnerVolumeSpecName "kube-api-access-jwctj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.335386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.335295    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnph\" (UniqueName: \"kubernetes.io/projected/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-kube-api-access-xfnph\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.335386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.335328    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.335386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.335338    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.335386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.335348    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwctj\" (UniqueName: \"kubernetes.io/projected/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a-kube-api-access-jwctj\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.335386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.335362    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dece7e-3005-418e-9c02-3dc0bf5daf6e-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.492787 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:19.492753    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5f1641-9264-4643-a750-a497ba1254da" path="/var/lib/kubelet/pods/2a5f1641-9264-4643-a750-a497ba1254da/volumes"
Oct 30 22:30:20.021053 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.021016    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"
Oct 30 22:30:20.021247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.021025    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644" event={"ID":"c7dece7e-3005-418e-9c02-3dc0bf5daf6e","Type":"ContainerDied","Data":"0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db"}
Oct 30 22:30:20.021247 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.021162    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfeeb1b4bc9d41fa07ebcd46a00b4707852633b87ba098af3e921580f23a5db"
Oct 30 22:30:20.022335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.022317    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"
Oct 30 22:30:20.022452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.022375    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj" event={"ID":"b540a89a-fbb7-4439-b07d-cbfa67ce7c8a","Type":"ContainerDied","Data":"3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f"}
Oct 30 22:30:20.022452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:20.022411    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fff7b83b6fb7b90bddc2709112faa3f11ec37419153d27a473f05ddbebe8a8f"
Oct 30 22:30:23.608979 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:23.608946    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:30:26.499773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.499736    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:30:26.500270 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.499958    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" podUID="c1717471-16bd-496b-b94a-1cb41698b09c" containerName="maas-api" containerID="cri-o://159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57" gracePeriod=30
Oct 30 22:30:26.777570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.777548    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:30:26.869332 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.869295    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:30:26.869592 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.869563    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy" containerID="cri-o://f2b0933be2487cfa468163a5d15e3f83698ae40caa7e3cb87c34004cf046bcb7" gracePeriod=30
Oct 30 22:30:26.929773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.925031    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\") pod \"c1717471-16bd-496b-b94a-1cb41698b09c\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") "
Oct 30 22:30:26.929773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.925110    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\") pod \"c1717471-16bd-496b-b94a-1cb41698b09c\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") "
Oct 30 22:30:26.929773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.925141    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") pod \"c1717471-16bd-496b-b94a-1cb41698b09c\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") "
Oct 30 22:30:26.929773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.925213    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\" (UniqueName: \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\") pod \"c1717471-16bd-496b-b94a-1cb41698b09c\" (UID: \"c1717471-16bd-496b-b94a-1cb41698b09c\") "
Oct 30 22:30:26.929773 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.926631    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca" pod "c1717471-16bd-496b-b94a-1cb41698b09c" (UID: "c1717471-16bd-496b-b94a-1cb41698b09c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:26.935907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.933777    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls" pod "c1717471-16bd-496b-b94a-1cb41698b09c" (UID: "c1717471-16bd-496b-b94a-1cb41698b09c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:26.936120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.936092    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs" pod "c1717471-16bd-496b-b94a-1cb41698b09c" (UID: "c1717471-16bd-496b-b94a-1cb41698b09c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:26.938590 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:26.938565    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4" pod "c1717471-16bd-496b-b94a-1cb41698b09c" (UID: "c1717471-16bd-496b-b94a-1cb41698b09c"). InnerVolumeSpecName "kube-api-access-cdps4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:27.027076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.026424    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:27.027076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.026501    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1717471-16bd-496b-b94a-1cb41698b09c-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:27.027076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.026518    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1717471-16bd-496b-b94a-1cb41698b09c-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:27.027076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.026533    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdps4\" (UniqueName: \"kubernetes.io/projected/c1717471-16bd-496b-b94a-1cb41698b09c-kube-api-access-cdps4\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:27.059599 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.059565    4295 generic.go:356] "Generic (PLEG): container finished" podID="c1717471-16bd-496b-b94a-1cb41698b09c" containerID="159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57" exitCode=0
Oct 30 22:30:27.059821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.059613    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"
Oct 30 22:30:27.059821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.059625    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" event={"ID":"c1717471-16bd-496b-b94a-1cb41698b09c","Type":"ContainerDied","Data":"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"}
Oct 30 22:30:27.059821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.059698    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4" event={"ID":"c1717471-16bd-496b-b94a-1cb41698b09c","Type":"ContainerDied","Data":"ccf2bcdb114c6d0646a2e10791242fe1a7bd93409bf0abef19487a78616c7b0e"}
Oct 30 22:30:27.059821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.059728    4295 scope.go:122] "RemoveContainer" containerID="159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"
Oct 30 22:30:27.069380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.069360    4295 scope.go:122] "RemoveContainer" containerID="159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"
Oct 30 22:30:27.069637 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:27.069619    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57\": container with ID starting with 159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57 not found: ID does not exist" containerID="159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"
Oct 30 22:30:27.069780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.069649    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57"} err="failed to get container status \"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57\": rpc error: code = NotFound desc = could not find container \"159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57\": container with ID starting with 159194f2bc82efba387c3d3a774e4fba9d68b3a4fdc3b3c9802e0d6f31135c57 not found: ID does not exist"
Oct 30 22:30:27.230205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.230164    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:30:27.244994 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.244962    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2182a5-6678bf5fc4-fd5q4"]
Oct 30 22:30:27.520340 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:27.520295    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1717471-16bd-496b-b94a-1cb41698b09c" path="/var/lib/kubelet/pods/c1717471-16bd-496b-b94a-1cb41698b09c/volumes"
Oct 30 22:30:32.085873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.085842    4295 generic.go:356] "Generic (PLEG): container finished" podID="22165712-d558-45ac-ba97-32d060e970d3" containerID="f2b0933be2487cfa468163a5d15e3f83698ae40caa7e3cb87c34004cf046bcb7" exitCode=0
Oct 30 22:30:32.086219 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.085910    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" event={"ID":"22165712-d558-45ac-ba97-32d060e970d3","Type":"ContainerDied","Data":"f2b0933be2487cfa468163a5d15e3f83698ae40caa7e3cb87c34004cf046bcb7"}
Oct 30 22:30:32.120487 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.120465    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:30:32.282200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282174    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282235    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282268    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282380 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282310    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282418    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282470    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282514    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282558    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282584    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:32.282614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282599    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:32.282614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282608    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\") pod \"22165712-d558-45ac-ba97-32d060e970d3\" (UID: \"22165712-d558-45ac-ba97-32d060e970d3\") "
Oct 30 22:30:32.282894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282730    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:32.282894 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282633    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:32.283062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282966    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22165712-d558-45ac-ba97-32d060e970d3-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.283062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282977    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:32.283062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.282983    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.283062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.283028    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.283062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.283043    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.285330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.285302    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:32.285458 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.285395    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "kube-api-access-x6mzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:32.285458 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.285442    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:32.285565 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.285470    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token" pod "22165712-d558-45ac-ba97-32d060e970d3" (UID: "22165712-d558-45ac-ba97-32d060e970d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:32.384335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.384301    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22165712-d558-45ac-ba97-32d060e970d3-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.384335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.384329    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mzk\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-kube-api-access-x6mzk\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.384335 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.384341    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22165712-d558-45ac-ba97-32d060e970d3-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.384567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.384351    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:32.384567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:32.384361    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22165712-d558-45ac-ba97-32d060e970d3-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.092704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.092643    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp" event={"ID":"22165712-d558-45ac-ba97-32d060e970d3","Type":"ContainerDied","Data":"d6fd738fd180bb4e81cc8d5e6a735e17fd9a928754360ce4b064b5bfb77f206b"}
Oct 30 22:30:33.092704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.092689    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"
Oct 30 22:30:33.093168 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.092732    4295 scope.go:122] "RemoveContainer" containerID="f2b0933be2487cfa468163a5d15e3f83698ae40caa7e3cb87c34004cf046bcb7"
Oct 30 22:30:33.318993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.318933    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:30:33.341010 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.340977    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp"]
Oct 30 22:30:33.493517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:33.493478    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22165712-d558-45ac-ba97-32d060e970d3" path="/var/lib/kubelet/pods/22165712-d558-45ac-ba97-32d060e970d3/volumes"
Oct 30 22:30:35.549674 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:35.549634    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"]
Oct 30 22:30:35.600500 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:35.600470    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d59c58-nlz9d"]
Oct 30 22:30:37.492942 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:37.492913    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80e475b-061a-4880-ad2e-949b7bd29297" path="/var/lib/kubelet/pods/b80e475b-061a-4880-ad2e-949b7bd29297/volumes"
Oct 30 22:30:37.865968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:37.865879    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 22:30:37.866145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:37.866078    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" containerID="cri-o://845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a" gracePeriod=30
Oct 30 22:30:37.893837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:37.893802    4295 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:37.893974 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:37.893864    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:43.119174 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.119149    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:30:43.145437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.145364    4295 generic.go:356] "Generic (PLEG): container finished" podID="d6736d70-17a0-4076-9119-6fd677801fe1" containerID="845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a" exitCode=0
Oct 30 22:30:43.145437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.145405    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"
Oct 30 22:30:43.145437 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.145413    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerDied","Data":"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"}
Oct 30 22:30:43.145637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.145464    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf" event={"ID":"d6736d70-17a0-4076-9119-6fd677801fe1","Type":"ContainerDied","Data":"92f09e078c767b7ca8d6593b4555e7f1270fab189816879e17a5322df19b935e"}
Oct 30 22:30:43.145637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.145488    4295 scope.go:122] "RemoveContainer" containerID="845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"
Oct 30 22:30:43.154214 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.154193    4295 scope.go:122] "RemoveContainer" containerID="abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"
Oct 30 22:30:43.162086 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.162066    4295 scope.go:122] "RemoveContainer" containerID="845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"
Oct 30 22:30:43.162347 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:43.162326    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a\": container with ID starting with 845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a not found: ID does not exist" containerID="845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"
Oct 30 22:30:43.162403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.162357    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a"} err="failed to get container status \"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a\": rpc error: code = NotFound desc = could not find container \"845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a\": container with ID starting with 845e638673cad2e6d0bb3d6ffa91cf43a95e3addc8b4e66dbb315fe1371f180a not found: ID does not exist"
Oct 30 22:30:43.162403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.162376    4295 scope.go:122] "RemoveContainer" containerID="abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"
Oct 30 22:30:43.162587 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:43.162571    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd\": container with ID starting with abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd not found: ID does not exist" containerID="abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"
Oct 30 22:30:43.162624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.162591    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd"} err="failed to get container status \"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd\": rpc error: code = NotFound desc = could not find container \"abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd\": container with ID starting with abbf0244f2f248385dce81f2edde06622d981d5df1342aea9ac274c8396b0abd not found: ID does not exist"
Oct 30 22:30:43.195061 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195040    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195107    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195133    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195270 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195159    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195270 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195177    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195270 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195220    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195419 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195281    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195491 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195453    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:43.195542 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195481    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:43.195542 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195468    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:43.195542 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195520    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195581    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\") pod \"d6736d70-17a0-4076-9119-6fd677801fe1\" (UID: \"d6736d70-17a0-4076-9119-6fd677801fe1\") "
Oct 30 22:30:43.195728 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195649    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:43.196031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.195997    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:43.196182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.196150    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.196182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.196180    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.196310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.196194    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.196310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.196209    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6736d70-17a0-4076-9119-6fd677801fe1-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.197456 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.197434    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "kube-api-access-2chtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:43.197562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.197509    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:43.197978 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.197954    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:43.198066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.197972    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy" pod "d6736d70-17a0-4076-9119-6fd677801fe1" (UID: "d6736d70-17a0-4076-9119-6fd677801fe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:43.296960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.296928    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.296960 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.296957    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chtr\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-kube-api-access-2chtr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.297167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.296972    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6736d70-17a0-4076-9119-6fd677801fe1-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.297167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.296984    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6736d70-17a0-4076-9119-6fd677801fe1-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.297167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.296997    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6736d70-17a0-4076-9119-6fd677801fe1-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:43.578962 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.578913    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 22:30:43.584211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:43.584183    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf"]
Oct 30 22:30:45.491602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.491564    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" path="/var/lib/kubelet/pods/d6736d70-17a0-4076-9119-6fd677801fe1/volumes"
Oct 30 22:30:45.811192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811106    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"]
Oct 30 22:30:45.811744 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811724    4295 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811745    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811767    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c7dece7e-3005-418e-9c02-3dc0bf5daf6e" containerName="revoke-keys"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811776    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7dece7e-3005-418e-9c02-3dc0bf5daf6e" containerName="revoke-keys"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811805    4295 cpu_manager.go:414] "containerMap: removing container" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811813    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811850    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" containerName="revoke-keys"
Oct 30 22:30:45.811865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811857    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" containerName="revoke-keys"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811877    4295 cpu_manager.go:414] "containerMap: removing container" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811885    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811911    4295 cpu_manager.go:414] "containerMap: removing container" podUID="c1717471-16bd-496b-b94a-1cb41698b09c" containerName="maas-api"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811918    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1717471-16bd-496b-b94a-1cb41698b09c" containerName="maas-api"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811929    4295 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811936    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811958    4295 cpu_manager.go:414] "containerMap: removing container" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.811967    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812142    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="22165712-d558-45ac-ba97-32d060e970d3" containerName="istio-proxy"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812175    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1717471-16bd-496b-b94a-1cb41698b09c" containerName="maas-api"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812185    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812204    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7dece7e-3005-418e-9c02-3dc0bf5daf6e" containerName="revoke-keys"
Oct 30 22:30:45.812223 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812214    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef297b82-0550-4038-bb69-fcf766aab2d5" containerName="cleanup"
Oct 30 22:30:45.812880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812231    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:45.812880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812241    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:45.812880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.812260    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" containerName="revoke-keys"
Oct 30 22:30:45.816968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.816929    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:45.874017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.873981    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"]
Oct 30 22:30:45.922482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.922450    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:45.922632 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.922493    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:30:45.930626 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.930598    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79k5\" (UniqueName: \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:45.930774 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.930644    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:45.930774 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:45.930752    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.032016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.031983    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79k5\" (UniqueName: \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.032192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.032027    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.032192 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.032047    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.032615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.032598    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.034375 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.034348    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.095330 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.095271    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79k5\" (UniqueName: \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.129251 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.129229    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:46.417982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.416732    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:30:46.417982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.416957    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy" containerID="cri-o://90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58" gracePeriod=30
Oct 30 22:30:46.554523 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:46.554493    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"]
Oct 30 22:30:47.167525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:47.167437    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p" event={"ID":"055218ec-0043-4936-b3d3-e0cc4ef310d3","Type":"ContainerStarted","Data":"ceb9262e9ce1ce9a72ea3685aad000c21a21a7da28eacabd06114b914cd53ef5"}
Oct 30 22:30:47.167525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:47.167489    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p" event={"ID":"055218ec-0043-4936-b3d3-e0cc4ef310d3","Type":"ContainerStarted","Data":"5e2f94ccbdefc602b04a06e5cea526ae4d31d5de22c26bcec52a76833a616c8d"}
Oct 30 22:30:47.292923 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:47.292871    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p" podStartSLOduration=2.292856327 podStartE2EDuration="2.292856327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:47.291633172 +0000 UTC m=+2449.934226952" watchObservedRunningTime="2026-10-30 22:30:47.292856327 +0000 UTC m=+2449.935450116"
Oct 30 22:30:48.173354 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:48.173315    4295 generic.go:356] "Generic (PLEG): container finished" podID="055218ec-0043-4936-b3d3-e0cc4ef310d3" containerID="ceb9262e9ce1ce9a72ea3685aad000c21a21a7da28eacabd06114b914cd53ef5" exitCode=0
Oct 30 22:30:48.173785 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:48.173392    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p" event={"ID":"055218ec-0043-4936-b3d3-e0cc4ef310d3","Type":"ContainerDied","Data":"ceb9262e9ce1ce9a72ea3685aad000c21a21a7da28eacabd06114b914cd53ef5"}
Oct 30 22:30:49.318843 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.318819    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:49.377634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.377605    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\") pod \"055218ec-0043-4936-b3d3-e0cc4ef310d3\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") "
Oct 30 22:30:49.377810 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.377793    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\" (UniqueName: \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\") pod \"055218ec-0043-4936-b3d3-e0cc4ef310d3\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") "
Oct 30 22:30:49.377884 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.377835    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\") pod \"055218ec-0043-4936-b3d3-e0cc4ef310d3\" (UID: \"055218ec-0043-4936-b3d3-e0cc4ef310d3\") "
Oct 30 22:30:49.378258 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.378234    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca" pod "055218ec-0043-4936-b3d3-e0cc4ef310d3" (UID: "055218ec-0043-4936-b3d3-e0cc4ef310d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:49.379721 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.379703    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca" pod "055218ec-0043-4936-b3d3-e0cc4ef310d3" (UID: "055218ec-0043-4936-b3d3-e0cc4ef310d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:49.380087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.380074    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5" pod "055218ec-0043-4936-b3d3-e0cc4ef310d3" (UID: "055218ec-0043-4936-b3d3-e0cc4ef310d3"). InnerVolumeSpecName "kube-api-access-b79k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:49.479691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.479582    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:49.479691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.479613    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b79k5\" (UniqueName: \"kubernetes.io/projected/055218ec-0043-4936-b3d3-e0cc4ef310d3-kube-api-access-b79k5\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:49.479691 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:49.479624    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055218ec-0043-4936-b3d3-e0cc4ef310d3-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:50.184621 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:50.184589    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"
Oct 30 22:30:50.184902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:50.184589    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p" event={"ID":"055218ec-0043-4936-b3d3-e0cc4ef310d3","Type":"ContainerDied","Data":"5e2f94ccbdefc602b04a06e5cea526ae4d31d5de22c26bcec52a76833a616c8d"}
Oct 30 22:30:50.184902 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:50.184757    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e2f94ccbdefc602b04a06e5cea526ae4d31d5de22c26bcec52a76833a616c8d"
Oct 30 22:30:50.242548 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:50.242516    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:30:50.293638 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:50.293577    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:30:51.671645 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.671621    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:30:51.802314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802235    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802286    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802321    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802362    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802399    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802504    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.802716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802536    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802622    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802717    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802760    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\") pod \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\" (UID: \"b75548ab-6bc9-4ba9-aafe-66661f5c280d\") "
Oct 30 22:30:51.802877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802770    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.802877 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802821    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:51.803080 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802954    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.803080 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.802967    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.803194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.803135    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.803194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.803153    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.803194 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.803168    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.803308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.803199    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.803308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.803212    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.804652 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.804633    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:51.805183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.805165    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.805243 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.805229    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.805282 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.805243    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4" pod "b75548ab-6bc9-4ba9-aafe-66661f5c280d" (UID: "b75548ab-6bc9-4ba9-aafe-66661f5c280d"). InnerVolumeSpecName "kube-api-access-fj4g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.904036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.904005    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.904036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.904030    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.904036 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.904040    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj4g4\" (UniqueName: \"kubernetes.io/projected/b75548ab-6bc9-4ba9-aafe-66661f5c280d-kube-api-access-fj4g4\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.904272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:51.904050    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75548ab-6bc9-4ba9-aafe-66661f5c280d-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:52.197054 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.197022    4295 generic.go:356] "Generic (PLEG): container finished" podID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerID="90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58" exitCode=0
Oct 30 22:30:52.197230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.197069    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"
Oct 30 22:30:52.197230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.197101    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" event={"ID":"b75548ab-6bc9-4ba9-aafe-66661f5c280d","Type":"ContainerDied","Data":"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"}
Oct 30 22:30:52.197230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.197156    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r" event={"ID":"b75548ab-6bc9-4ba9-aafe-66661f5c280d","Type":"ContainerDied","Data":"6e5c94a2f589881be980f3fd501b0d427828eb56327f148f384f3a06cc51ccee"}
Oct 30 22:30:52.197230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.197186    4295 scope.go:122] "RemoveContainer" containerID="90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"
Oct 30 22:30:52.206801 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.206783    4295 scope.go:122] "RemoveContainer" containerID="90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"
Oct 30 22:30:52.207083 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:52.207065    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58\": container with ID starting with 90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58 not found: ID does not exist" containerID="90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"
Oct 30 22:30:52.207148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.207104    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58"} err="failed to get container status \"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58\": rpc error: code = NotFound desc = could not find container \"90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58\": container with ID starting with 90328f5330e71ba2589c9c7c51f035ccfdcb76de5684c33ebbb2dbf5d962db58 not found: ID does not exist"
Oct 30 22:30:52.366709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.366674    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:30:52.417058 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:52.417027    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r"]
Oct 30 22:30:53.491401 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:53.491368    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" path="/var/lib/kubelet/pods/b75548ab-6bc9-4ba9-aafe-66661f5c280d/volumes"
Oct 30 22:30:53.607264 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:30:53.607228    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:30:57.057586 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057546    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"]
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057910    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057921    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057942    4295 cpu_manager.go:414] "containerMap: removing container" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057948    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057976    4295 cpu_manager.go:414] "containerMap: removing container" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057982    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057990    4295 cpu_manager.go:414] "containerMap: removing container" podUID="055218ec-0043-4936-b3d3-e0cc4ef310d3" containerName="revoke-keys"
Oct 30 22:30:57.058020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.057994    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055218ec-0043-4936-b3d3-e0cc4ef310d3" containerName="revoke-keys"
Oct 30 22:30:57.058267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.058045    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75548ab-6bc9-4ba9-aafe-66661f5c280d" containerName="istio-proxy"
Oct 30 22:30:57.058267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.058055    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5f1641-9264-4643-a750-a497ba1254da" containerName="cleanup"
Oct 30 22:30:57.058267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.058063    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6736d70-17a0-4076-9119-6fd677801fe1" containerName="istio-proxy"
Oct 30 22:30:57.058267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.058069    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="055218ec-0043-4936-b3d3-e0cc4ef310d3" containerName="revoke-keys"
Oct 30 22:30:57.060929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.060901    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.120369 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.120336    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"]
Oct 30 22:30:57.151172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.151145    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.151326 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.151206    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qkfv\" (UniqueName: \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.151370 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.151346    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.165130 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.165099    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:57.165261 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.165154    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:30:57.252184 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.252147    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qkfv\" (UniqueName: \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.252349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.252246    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.252349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.252311    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.252955 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.252927    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.254713 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.254694    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.318536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.318461    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qkfv\" (UniqueName: \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\") pod \"maas-api-revoke-sub-13718777-9chgv\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.372519 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.372488    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:30:57.752750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:57.752726    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"]
Oct 30 22:30:57.755529 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:30:57.755502    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a346b9_1772_44b1_a78d_19c15367e6c1.slice/crio-f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654 WatchSource:0}: Error finding container f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654: Status 404 returned error can't find the container with id f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654
Oct 30 22:30:58.227115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:58.227082    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv" event={"ID":"36a346b9-1772-44b1-a78d-19c15367e6c1","Type":"ContainerStarted","Data":"611fa7b9de4375a73881849685f8b5c038ba6d221278909fc23d099a345e7506"}
Oct 30 22:30:58.227525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:58.227132    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv" event={"ID":"36a346b9-1772-44b1-a78d-19c15367e6c1","Type":"ContainerStarted","Data":"f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654"}
Oct 30 22:30:58.347762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:58.347643    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv" podStartSLOduration=1.347629484 podStartE2EDuration="1.347629484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.346459664 +0000 UTC m=+2460.989053466" watchObservedRunningTime="2026-10-30 22:30:58.347629484 +0000 UTC m=+2460.990223272"
Oct 30 22:30:59.233990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:59.233956    4295 generic.go:356] "Generic (PLEG): container finished" podID="36a346b9-1772-44b1-a78d-19c15367e6c1" containerID="611fa7b9de4375a73881849685f8b5c038ba6d221278909fc23d099a345e7506" exitCode=0
Oct 30 22:30:59.234371 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:30:59.234045    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv" event={"ID":"36a346b9-1772-44b1-a78d-19c15367e6c1","Type":"ContainerDied","Data":"611fa7b9de4375a73881849685f8b5c038ba6d221278909fc23d099a345e7506"}
Oct 30 22:31:00.370337 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.370308    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:31:00.483606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.483576    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\") pod \"36a346b9-1772-44b1-a78d-19c15367e6c1\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") "
Oct 30 22:31:00.483802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.483700    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\" (UniqueName: \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\") pod \"36a346b9-1772-44b1-a78d-19c15367e6c1\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") "
Oct 30 22:31:00.483802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.483734    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\") pod \"36a346b9-1772-44b1-a78d-19c15367e6c1\" (UID: \"36a346b9-1772-44b1-a78d-19c15367e6c1\") "
Oct 30 22:31:00.484012 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.483986    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca" pod "36a346b9-1772-44b1-a78d-19c15367e6c1" (UID: "36a346b9-1772-44b1-a78d-19c15367e6c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:00.485883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.485854    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca" pod "36a346b9-1772-44b1-a78d-19c15367e6c1" (UID: "36a346b9-1772-44b1-a78d-19c15367e6c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:00.485988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.485956    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv" pod "36a346b9-1772-44b1-a78d-19c15367e6c1" (UID: "36a346b9-1772-44b1-a78d-19c15367e6c1"). InnerVolumeSpecName "kube-api-access-7qkfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:00.584833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.584794    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qkfv\" (UniqueName: \"kubernetes.io/projected/36a346b9-1772-44b1-a78d-19c15367e6c1-kube-api-access-7qkfv\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.584833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.584825    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.584833 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:00.584839    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a346b9-1772-44b1-a78d-19c15367e6c1-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.243900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:01.243869    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"
Oct 30 22:31:01.244075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:01.243911    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv" event={"ID":"36a346b9-1772-44b1-a78d-19c15367e6c1","Type":"ContainerDied","Data":"f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654"}
Oct 30 22:31:01.244075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:01.243956    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ccb8d7b5bd2f65d11fc7076103a058891eaeab095c4384e3f97a00bf1a3654"
Oct 30 22:31:05.261981 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.261937    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main" containerID="cri-o://ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61" gracePeriod=45
Oct 30 22:31:05.313379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.313322    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main" containerID="cri-o://22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648" gracePeriod=45
Oct 30 22:31:05.556381 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.556355    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:31:05.588310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.588288    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:31:05.638269 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638245    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638399 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638287    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638458 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638439    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\" (UniqueName: \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638534    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638609 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638575    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638606    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\") pod \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\" (UID: \"1f0b6b0d-bcbc-480d-9bcb-8923372d8035\") "
Oct 30 22:31:05.638883 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.638832    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.639195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.639132    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.639195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.639122    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.639195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.639164    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.640545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.640512    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:05.640545 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.640534    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.640835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.640819    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw" pod "1f0b6b0d-bcbc-480d-9bcb-8923372d8035" (UID: "1f0b6b0d-bcbc-480d-9bcb-8923372d8035"). InnerVolumeSpecName "kube-api-access-29wnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:05.740316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740277    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740338    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740439    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740494 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740488    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740517    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\" (UniqueName: \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740562    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\") pod \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\" (UID: \"8a824055-9e2c-46df-ae43-ae9c0bd169fd\") "
Oct 30 22:31:05.740688 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740588    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.740872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740704    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.740938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.740867    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.741043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741026    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741051    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741066    4295 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tmp-dir\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741080    4295 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-home\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741092    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741107 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741104    4295 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-model-cache\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741117    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.741310 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.741131    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wnw\" (UniqueName: \"kubernetes.io/projected/1f0b6b0d-bcbc-480d-9bcb-8923372d8035-kube-api-access-29wnw\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.742557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.742539    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:05.742629 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.742587    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.743180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.743160    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl" pod "8a824055-9e2c-46df-ae43-ae9c0bd169fd" (UID: "8a824055-9e2c-46df-ae43-ae9c0bd169fd"). InnerVolumeSpecName "kube-api-access-5kxrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:05.842225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.842148    4295 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a824055-9e2c-46df-ae43-ae9c0bd169fd-dshm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.842225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.842174    4295 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a824055-9e2c-46df-ae43-ae9c0bd169fd-tls-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.842225 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:05.842185    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxrl\" (UniqueName: \"kubernetes.io/projected/8a824055-9e2c-46df-ae43-ae9c0bd169fd-kube-api-access-5kxrl\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:06.266488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.266453    4295 generic.go:356] "Generic (PLEG): container finished" podID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerID="ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61" exitCode=0
Oct 30 22:31:06.267006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.266501    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"
Oct 30 22:31:06.267006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.266536    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" event={"ID":"1f0b6b0d-bcbc-480d-9bcb-8923372d8035","Type":"ContainerDied","Data":"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"}
Oct 30 22:31:06.267006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.266585    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894" event={"ID":"1f0b6b0d-bcbc-480d-9bcb-8923372d8035","Type":"ContainerDied","Data":"cb38babd51143e86ba080c1aa10ee4fb4fb459fb77c2bfcb48d3ab8ec9caa72a"}
Oct 30 22:31:06.267006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.266607    4295 scope.go:122] "RemoveContainer" containerID="ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"
Oct 30 22:31:06.267863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.267807    4295 generic.go:356] "Generic (PLEG): container finished" podID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerID="22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648" exitCode=0
Oct 30 22:31:06.267863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.267845    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"
Oct 30 22:31:06.267988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.267883    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" event={"ID":"8a824055-9e2c-46df-ae43-ae9c0bd169fd","Type":"ContainerDied","Data":"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"}
Oct 30 22:31:06.267988 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.267913    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz" event={"ID":"8a824055-9e2c-46df-ae43-ae9c0bd169fd","Type":"ContainerDied","Data":"abf0ede6eee5f47d1430e1c82c88ba8c29a6ef4177c61220deeba28469d94a2c"}
Oct 30 22:31:06.282808 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.282791    4295 scope.go:122] "RemoveContainer" containerID="ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"
Oct 30 22:31:06.283036 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:06.283019    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61\": container with ID starting with ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61 not found: ID does not exist" containerID="ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"
Oct 30 22:31:06.283094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.283043    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61"} err="failed to get container status \"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61\": rpc error: code = NotFound desc = could not find container \"ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61\": container with ID starting with ede87fc4d13ef57bb0079c9caffb3387ef24e333e48a7078831a5bb866cdec61 not found: ID does not exist"
Oct 30 22:31:06.283094 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.283059    4295 scope.go:122] "RemoveContainer" containerID="22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"
Oct 30 22:31:06.290506 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.290490    4295 scope.go:122] "RemoveContainer" containerID="22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"
Oct 30 22:31:06.290767 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:06.290752    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648\": container with ID starting with 22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648 not found: ID does not exist" containerID="22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"
Oct 30 22:31:06.290829 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.290773    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648"} err="failed to get container status \"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648\": rpc error: code = NotFound desc = could not find container \"22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648\": container with ID starting with 22dc4f5b25a23cc2803917695d2adb68990fb8fc3582b8cab5f878e5a615c648 not found: ID does not exist"
Oct 30 22:31:06.434615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.434583    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:31:06.440541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.440518    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-2-w3-8ca52a-kserve-6ddd99c467-sc894"]
Oct 30 22:31:06.553153 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.553077    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:31:06.558602 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:06.558580    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/premium-w3-8ca52a-kserve-586dc77669-pfqbz"]
Oct 30 22:31:07.491593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:07.491543    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" path="/var/lib/kubelet/pods/1f0b6b0d-bcbc-480d-9bcb-8923372d8035/volumes"
Oct 30 22:31:07.492063 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:07.492027    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" path="/var/lib/kubelet/pods/8a824055-9e2c-46df-ae43-ae9c0bd169fd/volumes"
Oct 30 22:31:14.293137 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:14.293052    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"]
Oct 30 22:31:14.343064 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:14.343027    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8ca52a-v2s4p"]
Oct 30 22:31:15.492087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:15.492056    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055218ec-0043-4936-b3d3-e0cc4ef310d3" path="/var/lib/kubelet/pods/055218ec-0043-4936-b3d3-e0cc4ef310d3/volumes"
Oct 30 22:31:22.977969 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:22.977936    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 22:31:22.978583 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:22.978200    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" containerID="cri-o://afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720" gracePeriod=30
Oct 30 22:31:23.607563 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:23.607528    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:31:28.227424 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.227398    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:31:28.277464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.277377    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 22:31:28.277694 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.277627    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" podUID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" containerName="manager" containerID="cri-o://59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6" gracePeriod=30
Oct 30 22:31:28.341075 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341043    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341102    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341136    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341323 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341299    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341340    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341421    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341425    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.341704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341538    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341584    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341610    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\") pod \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\" (UID: \"a1bab16b-a784-4c94-8ad7-a9bd39597bf6\") "
Oct 30 22:31:28.341704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341642    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341716    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341840    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.341963    4295 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.342010    4295 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-data\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.342025    4295 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-credential-socket\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.342035    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.342091 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.342039    4295 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istiod-ca-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.343559 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.343515    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "kube-api-access-hnzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:28.343656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.343593    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:28.343656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.343593    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:28.343910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.343889    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy" pod "a1bab16b-a784-4c94-8ad7-a9bd39597bf6" (UID: "a1bab16b-a784-4c94-8ad7-a9bd39597bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.367572 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.367547    4295 generic.go:356] "Generic (PLEG): container finished" podID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerID="afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720" exitCode=0
Oct 30 22:31:28.367696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.367598    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"
Oct 30 22:31:28.367696 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.367634    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerDied","Data":"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"}
Oct 30 22:31:28.367826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.367716    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" event={"ID":"a1bab16b-a784-4c94-8ad7-a9bd39597bf6","Type":"ContainerDied","Data":"cf6aaf436c5030abd3b6bf6357d2e19ee5974c6fa8aab4c89ff503220c98b247"}
Oct 30 22:31:28.367826 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.367751    4295 scope.go:122] "RemoveContainer" containerID="afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"
Oct 30 22:31:28.398948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.398922    4295 scope.go:122] "RemoveContainer" containerID="23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"
Oct 30 22:31:28.409505 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.409479    4295 scope.go:122] "RemoveContainer" containerID="afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"
Oct 30 22:31:28.409908 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:28.409886    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720\": container with ID starting with afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720 not found: ID does not exist" containerID="afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"
Oct 30 22:31:28.409989 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.409917    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720"} err="failed to get container status \"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720\": rpc error: code = NotFound desc = could not find container \"afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720\": container with ID starting with afa96761ceb6a3c3937b776a00b1685d08cb701c087cb65fe7c4ccab6da63720 not found: ID does not exist"
Oct 30 22:31:28.409989 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.409938    4295 scope.go:122] "RemoveContainer" containerID="23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"
Oct 30 22:31:28.410303 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:28.410281    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3\": container with ID starting with 23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3 not found: ID does not exist" containerID="23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"
Oct 30 22:31:28.410386 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.410312    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3"} err="failed to get container status \"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3\": rpc error: code = NotFound desc = could not find container \"23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3\": container with ID starting with 23aa52de0f314e7c53b0b900e16b5c46c00e4628bca34da5b98e15f0919324f3 not found: ID does not exist"
Oct 30 22:31:28.443213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.443181    4295 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-token\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.443213 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.443211    4295 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-podinfo\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.443444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.443221    4295 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-istio-envoy\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.443444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.443230    4295 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-workload-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.443444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.443240    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzw4\" (UniqueName: \"kubernetes.io/projected/a1bab16b-a784-4c94-8ad7-a9bd39597bf6-kube-api-access-hnzw4\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.518605 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.518578    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 22:31:28.644470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.644396    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\") pod \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") "
Oct 30 22:31:28.644624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.644534    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\") pod \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") "
Oct 30 22:31:28.644624 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.644564    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\" (UniqueName: \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\") pod \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\" (UID: \"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2\") "
Oct 30 22:31:28.647077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.646869    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs" pod "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" (UID: "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:28.647077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.646949    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert" pod "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" (UID: "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:28.647300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.647278    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v" pod "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" (UID: "1dd01aef-13c8-4346-a2cd-d3207ea4d7f2"). InnerVolumeSpecName "kube-api-access-5dz6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:28.649470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.649450    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 22:31:28.652582 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.652562    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz"]
Oct 30 22:31:28.745969 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.745948    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.745969 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.745970    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dz6v\" (UniqueName: \"kubernetes.io/projected/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-kube-api-access-5dz6v\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.746126 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:28.745982    4295 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2-webhook-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:29.217023 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.216977    4295 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:31:29.217199 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.217055    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:29.373282 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.373192    4295 generic.go:356] "Generic (PLEG): container finished" podID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" containerID="59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6" exitCode=0
Oct 30 22:31:29.373282 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.373249    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9"
Oct 30 22:31:29.373798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.373278    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" event={"ID":"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2","Type":"ContainerDied","Data":"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"}
Oct 30 22:31:29.373798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.373362    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-d8xc9" event={"ID":"1dd01aef-13c8-4346-a2cd-d3207ea4d7f2","Type":"ContainerDied","Data":"24e9d37ad062f496981c0428bf4513d976ff88d91bc7d760bf256a499b5e2d64"}
Oct 30 22:31:29.373798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.373395    4295 scope.go:122] "RemoveContainer" containerID="59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"
Oct 30 22:31:29.382788 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.382771    4295 scope.go:122] "RemoveContainer" containerID="59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"
Oct 30 22:31:29.383062 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:29.383044    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6\": container with ID starting with 59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6 not found: ID does not exist" containerID="59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"
Oct 30 22:31:29.383139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.383069    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6"} err="failed to get container status \"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6\": rpc error: code = NotFound desc = could not find container \"59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6\": container with ID starting with 59cb677d42d196313f091817cdb6bdbe84adb1b1c2f2ba814f54a1326cc90fd6 not found: ID does not exist"
Oct 30 22:31:29.491808 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.491776    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" path="/var/lib/kubelet/pods/a1bab16b-a784-4c94-8ad7-a9bd39597bf6/volumes"
Oct 30 22:31:29.546167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.546140    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 22:31:29.552920 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:29.552898    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-d8xc9"]
Oct 30 22:31:30.438438 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438407    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438735    4295 cpu_manager.go:414] "containerMap: removing container" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438746    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438760    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" containerName="manager"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438764    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" containerName="manager"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438770    4295 cpu_manager.go:414] "containerMap: removing container" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438776    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438787    4295 cpu_manager.go:414] "containerMap: removing container" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438791    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438797    4295 cpu_manager.go:414] "containerMap: removing container" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438801    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438809    4295 cpu_manager.go:414] "containerMap: removing container" podUID="36a346b9-1772-44b1-a78d-19c15367e6c1" containerName="revoke-keys"
Oct 30 22:31:30.438835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438814    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a346b9-1772-44b1-a78d-19c15367e6c1" containerName="revoke-keys"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438891    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a824055-9e2c-46df-ae43-ae9c0bd169fd" containerName="main"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438901    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" containerName="manager"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438906    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0b6b0d-bcbc-480d-9bcb-8923372d8035" containerName="main"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438913    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438918    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bab16b-a784-4c94-8ad7-a9bd39597bf6" containerName="istio-proxy"
Oct 30 22:31:30.439200 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.438925    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a346b9-1772-44b1-a78d-19c15367e6c1" containerName="revoke-keys"
Oct 30 22:31:30.443067 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.443051    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.547635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.547598    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 22:31:30.547635 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.547609    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-24lvx\""
Oct 30 22:31:30.547865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.547655    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 22:31:30.547865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.547703    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 22:31:30.549683 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.549638    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:31:30.562465 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.562437    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.562593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.562574    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjthx\" (UniqueName: \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.562649 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.562638    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.664062 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.664026    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.664256 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.664122    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjthx\" (UniqueName: \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.664256 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.664175    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.666598 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.666575    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.666733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.666598    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.729625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.729559    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjthx\" (UniqueName: \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\") pod \"maas-controller-5fb4f557f9-j55j4\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") " pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:30.755280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:30.755256    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:31.133300 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:31.133272    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:31:31.135517 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:31:31.135479    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f15025_a874_42a9_ae4a_dc01916db22f.slice/crio-65ef2d13b2bb60c5619dca1d2498e8037d3bc12d95756ee93a1b916c22cfc191 WatchSource:0}: Error finding container 65ef2d13b2bb60c5619dca1d2498e8037d3bc12d95756ee93a1b916c22cfc191: Status 404 returned error can't find the container with id 65ef2d13b2bb60c5619dca1d2498e8037d3bc12d95756ee93a1b916c22cfc191
Oct 30 22:31:31.137032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:31.137011    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:31:31.385704 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:31.385595    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" event={"ID":"57f15025-a874-42a9-ae4a-dc01916db22f","Type":"ContainerStarted","Data":"65ef2d13b2bb60c5619dca1d2498e8037d3bc12d95756ee93a1b916c22cfc191"}
Oct 30 22:31:31.492464 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:31.492424    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd01aef-13c8-4346-a2cd-d3207ea4d7f2" path="/var/lib/kubelet/pods/1dd01aef-13c8-4346-a2cd-d3207ea4d7f2/volumes"
Oct 30 22:31:32.391463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:32.391382    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" event={"ID":"57f15025-a874-42a9-ae4a-dc01916db22f","Type":"ContainerStarted","Data":"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"}
Oct 30 22:31:32.391604 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:32.391507    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:32.517351 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:32.517288    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" podStartSLOduration=1.6189715009999999 podStartE2EDuration="2.517275543s" totalImagesPullingTime="898.304042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:32.515501014 +0000 UTC m=+2495.158094803" watchObservedRunningTime="2026-10-30 22:31:32.517275543 +0000 UTC m=+2495.159869331"
Oct 30 22:31:43.410476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:43.410445    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:31:51.622547 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.622512    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"]
Oct 30 22:31:51.626044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.626027    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.681396 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.681372    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:51.730287 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.730259    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:31:51.732220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.732196    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"]
Oct 30 22:31:51.800867 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.800836    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.801031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.800900    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.801099 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.801022    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2tt\" (UniqueName: \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.901743 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.901627    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.901904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.901747    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.901904 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.901778    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2tt\" (UniqueName: \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.902274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.902250    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.904122 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.904101    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:51.966755 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:51.966719    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2tt\" (UniqueName: \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\") pod \"maas-api-revoke-sub-85237e45-59k6b\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:52.237925 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:52.237893    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:52.613028 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:52.613005    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"]
Oct 30 22:31:52.614819 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:31:52.614789    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb697057_4dbb_4f83_980e_f840da486421.slice/crio-0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c WatchSource:0}: Error finding container 0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c: Status 404 returned error can't find the container with id 0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c
Oct 30 22:31:53.537080 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:53.537043    4295 generic.go:356] "Generic (PLEG): container finished" podID="eb697057-4dbb-4f83-980e-f840da486421" containerID="fd8d96d0902167ac69b6d1fee24062bd2e58d5331b805d7184a5371b18735892" exitCode=0
Oct 30 22:31:53.537482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:53.537117    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b" event={"ID":"eb697057-4dbb-4f83-980e-f840da486421","Type":"ContainerDied","Data":"fd8d96d0902167ac69b6d1fee24062bd2e58d5331b805d7184a5371b18735892"}
Oct 30 22:31:53.537482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:53.537154    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b" event={"ID":"eb697057-4dbb-4f83-980e-f840da486421","Type":"ContainerStarted","Data":"0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c"}
Oct 30 22:31:53.607576 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:31:53.607498    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:31:54.670373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.670350    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:54.826975 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.826897    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\") pod \"eb697057-4dbb-4f83-980e-f840da486421\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") "
Oct 30 22:31:54.827148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.827048    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\" (UniqueName: \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\") pod \"eb697057-4dbb-4f83-980e-f840da486421\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") "
Oct 30 22:31:54.827148 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.827083    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\") pod \"eb697057-4dbb-4f83-980e-f840da486421\" (UID: \"eb697057-4dbb-4f83-980e-f840da486421\") "
Oct 30 22:31:54.827484 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.827461    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca" pod "eb697057-4dbb-4f83-980e-f840da486421" (UID: "eb697057-4dbb-4f83-980e-f840da486421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:54.829160 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.829135    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt" pod "eb697057-4dbb-4f83-980e-f840da486421" (UID: "eb697057-4dbb-4f83-980e-f840da486421"). InnerVolumeSpecName "kube-api-access-7n2tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:54.829249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.829158    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca" pod "eb697057-4dbb-4f83-980e-f840da486421" (UID: "eb697057-4dbb-4f83-980e-f840da486421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:54.928229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.928192    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb697057-4dbb-4f83-980e-f840da486421-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:54.928229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.928222    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb697057-4dbb-4f83-980e-f840da486421-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:54.928229 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:54.928233    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2tt\" (UniqueName: \"kubernetes.io/projected/eb697057-4dbb-4f83-980e-f840da486421-kube-api-access-7n2tt\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:55.547136 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:55.547103    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"
Oct 30 22:31:55.547311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:55.547108    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b" event={"ID":"eb697057-4dbb-4f83-980e-f840da486421","Type":"ContainerDied","Data":"0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c"}
Oct 30 22:31:55.547311 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:31:55.547226    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0095a69b3a62ca322a9a66e88112859eb3fc9dd6bf3a8ae877a4d0d91c664f3c"
Oct 30 22:32:04.295763 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.295725    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:32:04.296191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.295965    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" podUID="57f15025-a874-42a9-ae4a-dc01916db22f" containerName="manager" containerID="cri-o://d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d" gracePeriod=30
Oct 30 22:32:04.545860 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.545805    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:32:04.587848 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.587817    4295 generic.go:356] "Generic (PLEG): container finished" podID="57f15025-a874-42a9-ae4a-dc01916db22f" containerID="d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d" exitCode=0
Oct 30 22:32:04.587993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.587855    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-j55j4"
Oct 30 22:32:04.587993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.587862    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" event={"ID":"57f15025-a874-42a9-ae4a-dc01916db22f","Type":"ContainerDied","Data":"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"}
Oct 30 22:32:04.587993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.587907    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-j55j4" event={"ID":"57f15025-a874-42a9-ae4a-dc01916db22f","Type":"ContainerDied","Data":"65ef2d13b2bb60c5619dca1d2498e8037d3bc12d95756ee93a1b916c22cfc191"}
Oct 30 22:32:04.587993 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.587928    4295 scope.go:122] "RemoveContainer" containerID="d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"
Oct 30 22:32:04.596125 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.596109    4295 scope.go:122] "RemoveContainer" containerID="d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"
Oct 30 22:32:04.596415 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:32:04.596397    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d\": container with ID starting with d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d not found: ID does not exist" containerID="d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"
Oct 30 22:32:04.596462 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.596424    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d"} err="failed to get container status \"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d\": rpc error: code = NotFound desc = could not find container \"d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d\": container with ID starting with d6a2821c56eea1e1f4bfc720c902368ea1a3ae8b5c637252226e745533816c9d not found: ID does not exist"
Oct 30 22:32:04.616540 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.616516    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\") pod \"57f15025-a874-42a9-ae4a-dc01916db22f\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") "
Oct 30 22:32:04.616623 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.616555    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\") pod \"57f15025-a874-42a9-ae4a-dc01916db22f\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") "
Oct 30 22:32:04.616709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.616696    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\" (UniqueName: \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\") pod \"57f15025-a874-42a9-ae4a-dc01916db22f\" (UID: \"57f15025-a874-42a9-ae4a-dc01916db22f\") "
Oct 30 22:32:04.618699 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.618641    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert" pod "57f15025-a874-42a9-ae4a-dc01916db22f" (UID: "57f15025-a874-42a9-ae4a-dc01916db22f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:04.618802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.618779    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx" pod "57f15025-a874-42a9-ae4a-dc01916db22f" (UID: "57f15025-a874-42a9-ae4a-dc01916db22f"). InnerVolumeSpecName "kube-api-access-cjthx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:04.618802 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.618779    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs" pod "57f15025-a874-42a9-ae4a-dc01916db22f" (UID: "57f15025-a874-42a9-ae4a-dc01916db22f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:04.717793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.717762    4295 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-webhook-cert\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.717793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.717791    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f15025-a874-42a9-ae4a-dc01916db22f-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.717980 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:04.717803    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjthx\" (UniqueName: \"kubernetes.io/projected/57f15025-a874-42a9-ae4a-dc01916db22f-kube-api-access-cjthx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:05.012562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:05.012527    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:32:05.018648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:05.018622    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-j55j4"]
Oct 30 22:32:05.491078 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:05.491046    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f15025-a874-42a9-ae4a-dc01916db22f" path="/var/lib/kubelet/pods/57f15025-a874-42a9-ae4a-dc01916db22f/volumes"
Oct 30 22:32:06.441725 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.441695    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-t8dxn"]
Oct 30 22:32:06.442040 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442027    4295 cpu_manager.go:414] "containerMap: removing container" podUID="eb697057-4dbb-4f83-980e-f840da486421" containerName="revoke-keys"
Oct 30 22:32:06.442092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442042    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb697057-4dbb-4f83-980e-f840da486421" containerName="revoke-keys"
Oct 30 22:32:06.442092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442064    4295 cpu_manager.go:414] "containerMap: removing container" podUID="57f15025-a874-42a9-ae4a-dc01916db22f" containerName="manager"
Oct 30 22:32:06.442092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442069    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f15025-a874-42a9-ae4a-dc01916db22f" containerName="manager"
Oct 30 22:32:06.442206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442124    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb697057-4dbb-4f83-980e-f840da486421" containerName="revoke-keys"
Oct 30 22:32:06.442206 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.442135    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f15025-a874-42a9-ae4a-dc01916db22f" containerName="manager"
Oct 30 22:32:06.446425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.446408    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.531778 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.531751    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-metrics-certs\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.532121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.531785    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw5k\" (UniqueName: \"kubernetes.io/projected/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-kube-api-access-rzw5k\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.532121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.531891    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-webhook-cert\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.550928 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.550905    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 22:32:06.551079 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.551064    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 22:32:06.551145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.551065    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-24lvx\""
Oct 30 22:32:06.551145 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.551123    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 22:32:06.553552 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.553532    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-t8dxn"]
Oct 30 22:32:06.633096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.633066    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-webhook-cert\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.633271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.633131    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-metrics-certs\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.633271 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.633157    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw5k\" (UniqueName: \"kubernetes.io/projected/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-kube-api-access-rzw5k\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.635765 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.635744    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-metrics-certs\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.635859 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.635792    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-webhook-cert\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.696767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.696713    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw5k\" (UniqueName: \"kubernetes.io/projected/d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f-kube-api-access-rzw5k\") pod \"maas-controller-5fb4f557f9-t8dxn\" (UID: \"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f\") " pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:06.758167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:06.758147    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:07.133709 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:07.133685    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb4f557f9-t8dxn"]
Oct 30 22:32:07.135344 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:32:07.135314    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a4caa9_4234_4a62_9e4f_4bf9c1c8571f.slice/crio-8d03e21a5190345a6a5834064453054c6fff22f56d29f94ce16cda2ab27dd8f9 WatchSource:0}: Error finding container 8d03e21a5190345a6a5834064453054c6fff22f56d29f94ce16cda2ab27dd8f9: Status 404 returned error can't find the container with id 8d03e21a5190345a6a5834064453054c6fff22f56d29f94ce16cda2ab27dd8f9
Oct 30 22:32:07.603348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:07.603312    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn" event={"ID":"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f","Type":"ContainerStarted","Data":"8d03e21a5190345a6a5834064453054c6fff22f56d29f94ce16cda2ab27dd8f9"}
Oct 30 22:32:08.608754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:08.608719    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn" event={"ID":"d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f","Type":"ContainerStarted","Data":"fc7b6d9ebb88124cf50a9cf164726c3cdfc62970a591a8c0515d0d600cbbf3fc"}
Oct 30 22:32:08.609155 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:08.608832    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:08.731292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:08.731242    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn" podStartSLOduration=1.953308174 podStartE2EDuration="2.731226339s" totalImagesPullingTime="777.918165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:08.730263159 +0000 UTC m=+2531.372856951" watchObservedRunningTime="2026-10-30 22:32:08.731226339 +0000 UTC m=+2531.373820127"
Oct 30 22:32:19.616628 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:19.616599    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb4f557f9-t8dxn"
Oct 30 22:32:23.607696 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:32:23.607633    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:32:53.776626 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:32:53.776584    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:32:59.993729 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:59.993698    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"]
Oct 30 22:32:59.998172 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:32:59.998156    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.050874 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.050840    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.051035 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.050899    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.051035 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.050977    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djqx\" (UniqueName: \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.058828 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.058797    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"]
Oct 30 22:33:00.106316 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.106279    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:00.106471 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.106323    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:33:00.152049 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.152021    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.152232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.152064    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djqx\" (UniqueName: \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.152232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.152154    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.152747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.152725    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.154596 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.154572    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.218840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.218811    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djqx\" (UniqueName: \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\") pod \"maas-api-revoke-sub-c1e271a9-mhxbf\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.310654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.310588    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:00.685322 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.685297    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"]
Oct 30 22:33:00.687288 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:33:00.687256    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026de0a4_88fd_4a20_b558_28d8d8f5241a.slice/crio-5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db WatchSource:0}: Error finding container 5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db: Status 404 returned error can't find the container with id 5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db
Oct 30 22:33:00.853533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:00.853503    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf" event={"ID":"026de0a4-88fd-4a20-b558-28d8d8f5241a","Type":"ContainerStarted","Data":"5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db"}
Oct 30 22:33:01.860131 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:01.860040    4295 generic.go:356] "Generic (PLEG): container finished" podID="026de0a4-88fd-4a20-b558-28d8d8f5241a" containerID="8c3db75fbab7fa9bcba3136415b1446b2d22bac560443eb3e58c36627fe8f102" exitCode=0
Oct 30 22:33:01.860499 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:01.860125    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf" event={"ID":"026de0a4-88fd-4a20-b558-28d8d8f5241a","Type":"ContainerDied","Data":"8c3db75fbab7fa9bcba3136415b1446b2d22bac560443eb3e58c36627fe8f102"}
Oct 30 22:33:03.001355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.001328    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:03.080304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.080276    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\" (UniqueName: \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\") pod \"026de0a4-88fd-4a20-b558-28d8d8f5241a\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") "
Oct 30 22:33:03.080427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.080339    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\") pod \"026de0a4-88fd-4a20-b558-28d8d8f5241a\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") "
Oct 30 22:33:03.080427 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.080394    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\") pod \"026de0a4-88fd-4a20-b558-28d8d8f5241a\" (UID: \"026de0a4-88fd-4a20-b558-28d8d8f5241a\") "
Oct 30 22:33:03.080767 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.080747    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca" pod "026de0a4-88fd-4a20-b558-28d8d8f5241a" (UID: "026de0a4-88fd-4a20-b558-28d8d8f5241a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:03.082415 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.082395    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca" pod "026de0a4-88fd-4a20-b558-28d8d8f5241a" (UID: "026de0a4-88fd-4a20-b558-28d8d8f5241a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:03.082492 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.082472    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx" pod "026de0a4-88fd-4a20-b558-28d8d8f5241a" (UID: "026de0a4-88fd-4a20-b558-28d8d8f5241a"). InnerVolumeSpecName "kube-api-access-8djqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:03.181391 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.181369    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djqx\" (UniqueName: \"kubernetes.io/projected/026de0a4-88fd-4a20-b558-28d8d8f5241a-kube-api-access-8djqx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:03.181533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.181395    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:03.181533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.181405    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026de0a4-88fd-4a20-b558-28d8d8f5241a-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:03.870541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.870464    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"
Oct 30 22:33:03.870541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.870484    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf" event={"ID":"026de0a4-88fd-4a20-b558-28d8d8f5241a","Type":"ContainerDied","Data":"5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db"}
Oct 30 22:33:03.870541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:33:03.870530    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5def88c41ade6c5f0eba63297c274352a76c7a3041aca29d167ed2e9d9c393db"
Oct 30 22:33:23.606731 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:33:23.606689    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:33:53.607393 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:33:53.607343    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:34:23.607534 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:34:23.607454    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:34:44.063127 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:34:44.063092    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"]
Oct 30 22:34:44.113285 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:34:44.113243    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70407806-nkl8v"]
Oct 30 22:34:45.492738 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:34:45.492704    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b823cba-638a-4058-9635-fa8604d96c87" path="/var/lib/kubelet/pods/5b823cba-638a-4058-9635-fa8604d96c87/volumes"
Oct 30 22:34:53.625545 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:34:53.625502    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:34:56.142698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:34:56.141766    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-whxxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:56.142698 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:34:56.141830    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj" podUID="310492ba-b2e0-4610-90f6-6fb5d5deb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:35:15.062201 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:15.062162    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"]
Oct 30 22:35:15.112381 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:15.112357    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32203fc5-97gp9"]
Oct 30 22:35:15.492575 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:15.492543    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd8804d-5d51-4a05-94d5-e3642d47b1a0" path="/var/lib/kubelet/pods/cdd8804d-5d51-4a05-94d5-e3642d47b1a0/volumes"
Oct 30 22:35:19.072746 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.072712    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"]
Oct 30 22:35:19.123385 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.123345    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"]
Oct 30 22:35:19.124637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.124622    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c74b376-qq644"]
Oct 30 22:35:19.126039 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.126019    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a30b195-ggfsj"]
Oct 30 22:35:19.492600 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.492568    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b540a89a-fbb7-4439-b07d-cbfa67ce7c8a" path="/var/lib/kubelet/pods/b540a89a-fbb7-4439-b07d-cbfa67ce7c8a/volumes"
Oct 30 22:35:19.492990 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:19.492976    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dece7e-3005-418e-9c02-3dc0bf5daf6e" path="/var/lib/kubelet/pods/c7dece7e-3005-418e-9c02-3dc0bf5daf6e/volumes"
Oct 30 22:35:23.608282 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:35:23.608238    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:35:53.608000 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:35:53.607959    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:35:58.412024 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:35:58.411988    4295 scope.go:122] "RemoveContainer" containerID="65fea916606834e006106cb02e1a4ed076e0192a68ffdbda6f6be4a5211184e1"
Oct 30 22:36:01.075476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:01.075425    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"]
Oct 30 22:36:01.113535 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:01.113500    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13718777-9chgv"]
Oct 30 22:36:01.502968 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:01.502934    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a346b9-1772-44b1-a78d-19c15367e6c1" path="/var/lib/kubelet/pods/36a346b9-1772-44b1-a78d-19c15367e6c1/volumes"
Oct 30 22:36:23.607516 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:36:23.607469    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:36:53.607466 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:36:53.607424    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:36:55.058403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:55.058363    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"]
Oct 30 22:36:55.109612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:55.109571    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-59k6b"]
Oct 30 22:36:55.491529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:55.491495    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb697057-4dbb-4f83-980e-f840da486421" path="/var/lib/kubelet/pods/eb697057-4dbb-4f83-980e-f840da486421/volumes"
Oct 30 22:36:58.433318 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.433287    4295 scope.go:122] "RemoveContainer" containerID="611fa7b9de4375a73881849685f8b5c038ba6d221278909fc23d099a345e7506"
Oct 30 22:36:58.441921 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.441901    4295 scope.go:122] "RemoveContainer" containerID="ceb9262e9ce1ce9a72ea3685aad000c21a21a7da28eacabd06114b914cd53ef5"
Oct 30 22:36:58.449565 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.449548    4295 scope.go:122] "RemoveContainer" containerID="22dd10b894b4eb64c86afe62853d62595c6babd367fae81f05c8a6a4eb8b7df1"
Oct 30 22:36:58.457005 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.456977    4295 scope.go:122] "RemoveContainer" containerID="ec7c08a25c29c5a9ceb7213353ac820acf972a6be382e07bff63d6e838e177b9"
Oct 30 22:36:58.463934 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.463915    4295 scope.go:122] "RemoveContainer" containerID="a3975f1f420bdeb6ad14acb28356534208745bc43e4d030a0e06833b2fbb2d5e"
Oct 30 22:36:58.470795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:36:58.470778    4295 scope.go:122] "RemoveContainer" containerID="a77adfc3851ca9d2d1f46ed2af99d51dc80af8504dac95fcd00689371fb8d885"
Oct 30 22:37:12.663873 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:37:12.663777    4295 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:37:12.664779 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:37:12.664742    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57" podUID="dafe4d11-269b-499c-a118-520bdf759084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:37:23.607646 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:37:23.607609    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:37:53.607281 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:37:53.607245    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:37:58.504718 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:37:58.504683    4295 scope.go:122] "RemoveContainer" containerID="fd8d96d0902167ac69b6d1fee24062bd2e58d5331b805d7184a5371b18735892"
Oct 30 22:38:04.069533 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:38:04.069493    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"]
Oct 30 22:38:04.119901 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:38:04.119860    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mhxbf"]
Oct 30 22:38:05.491789 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:38:05.491753    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026de0a4-88fd-4a20-b558-28d8d8f5241a" path="/var/lib/kubelet/pods/026de0a4-88fd-4a20-b558-28d8d8f5241a/volumes"
Oct 30 22:38:23.607685 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:38:23.607629    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:38:53.607398 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:38:53.607360    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:39:23.607801 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:39:23.607764    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:39:25.717043 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.716986    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:39:25.717572 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.717508    4295 cpu_manager.go:414] "containerMap: removing container" podUID="026de0a4-88fd-4a20-b558-28d8d8f5241a" containerName="revoke-keys"
Oct 30 22:39:25.717572 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.717526    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026de0a4-88fd-4a20-b558-28d8d8f5241a" containerName="revoke-keys"
Oct 30 22:39:25.717654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.717635    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="026de0a4-88fd-4a20-b558-28d8d8f5241a" containerName="revoke-keys"
Oct 30 22:39:25.722864 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.722798    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.776032 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.776001    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:39:25.797690 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.797638    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.797870 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.797809    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ftb\" (UniqueName: \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.824488 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.824464    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-63518488-dockercfg-v75zz\""
Oct 30 22:39:25.824625 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.824504    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ca3b17\""
Oct 30 22:39:25.898515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.898482    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ftb\" (UniqueName: \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.898715 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.898565    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.899244 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.899223    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:25.962570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:25.962544    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ftb\" (UniqueName: \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\") pod \"payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:26.044724 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.044625    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:26.417748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.417725    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:39:26.419951 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:39:26.419932    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e82f4c_4d5e_4ef4_87b5_cd34a6e1947a.slice/crio-3509b9ce4438c5502754873f44b4d631575869eb3ea155d998f33ba64cd573ae WatchSource:0}: Error finding container 3509b9ce4438c5502754873f44b4d631575869eb3ea155d998f33ba64cd573ae: Status 404 returned error can't find the container with id 3509b9ce4438c5502754873f44b4d631575869eb3ea155d998f33ba64cd573ae
Oct 30 22:39:26.826508 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.826477    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" event={"ID":"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a","Type":"ContainerStarted","Data":"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"}
Oct 30 22:39:26.826909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.826523    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" event={"ID":"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a","Type":"ContainerStarted","Data":"3509b9ce4438c5502754873f44b4d631575869eb3ea155d998f33ba64cd573ae"}
Oct 30 22:39:26.826909 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.826552    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:26.947771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:26.947700    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" podStartSLOduration=1.947686985 podStartE2EDuration="1.947686985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:26.946346553 +0000 UTC m=+2969.588940343" watchObservedRunningTime="2026-10-30 22:39:26.947686985 +0000 UTC m=+2969.590280773"
Oct 30 22:39:28.273594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.273558    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:39:28.276863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.276845    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.331879 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.331854    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ca3b17-serving-cert\""
Oct 30 22:39:28.381571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.381544    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:39:28.381571 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.381549    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\""
Oct 30 22:39:28.383240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.383221    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:39:28.432031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.431987    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.432215 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.432037    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.432290 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.432241    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.432349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.432316    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.533769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.533642    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.533769 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.533765    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.534006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.533799    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.534006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.533898    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.534977 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:39:28.534953    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-serving-cert: secret "maas-api-e2e-worker-single-main-ca3b17-serving-cert" not found
Oct 30 22:39:28.535234 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:39:28.535211    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls podName:3d207c0c-4058-4eae-bcfd-3839d6c7ba37 nodeName:}" failed. No retries permitted until 2026-10-30 22:39:29.035187058 +0000 UTC m=+2971.677780827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls") pod "maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" (UID: "3d207c0c-4058-4eae-bcfd-3839d6c7ba37") : secret "maas-api-e2e-worker-single-main-ca3b17-serving-cert" not found
Oct 30 22:39:28.535984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.535960    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.544417 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.544391    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:28.598431 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:28.598397    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:29.039273 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.039239    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:29.041885 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.041860    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:29.190009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.189982    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:29.565799 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.565774    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:39:29.567788 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:39:29.567760    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d207c0c_4058_4eae_bcfd_3839d6c7ba37.slice/crio-c9aa03f78e6238e7f28c3dd594b3c84151c68c2dd9988e44bfdaef0febb226f3 WatchSource:0}: Error finding container c9aa03f78e6238e7f28c3dd594b3c84151c68c2dd9988e44bfdaef0febb226f3: Status 404 returned error can't find the container with id c9aa03f78e6238e7f28c3dd594b3c84151c68c2dd9988e44bfdaef0febb226f3
Oct 30 22:39:29.569329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.569309    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:39:29.840089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:29.840015    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" event={"ID":"3d207c0c-4058-4eae-bcfd-3839d6c7ba37","Type":"ContainerStarted","Data":"c9aa03f78e6238e7f28c3dd594b3c84151c68c2dd9988e44bfdaef0febb226f3"}
Oct 30 22:39:30.845895 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:30.845802    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" event={"ID":"3d207c0c-4058-4eae-bcfd-3839d6c7ba37","Type":"ContainerStarted","Data":"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"}
Oct 30 22:39:30.846260 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:30.845913    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:30.968634 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:30.968584    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" podStartSLOduration=2.081416241 podStartE2EDuration="2.968570765s" totalImagesPullingTime="887.154524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:30.967526131 +0000 UTC m=+2973.610138668" watchObservedRunningTime="2026-10-30 22:39:30.968570765 +0000 UTC m=+2973.611164553"
Oct 30 22:39:36.854865 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:36.854836    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:39:37.834307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:37.834280    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:39:53.606612 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:39:53.606564    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:39:58.536793 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:39:58.536757    4295 scope.go:122] "RemoveContainer" containerID="8c3db75fbab7fa9bcba3136415b1446b2d22bac560443eb3e58c36627fe8f102"
Oct 30 22:40:23.606854 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:40:23.606778    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:40:53.607548 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:40:53.607512    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:40:57.987020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:40:57.986987    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:41:01.242593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:01.242546    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:41:01.243120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:01.242833    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerName="maas-api" containerID="cri-o://afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2" gracePeriod=30
Oct 30 22:41:01.850230 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:01.850184    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.139:8443/health\": dial tcp 10.133.0.139:8443: connect: connection refused"
Oct 30 22:41:02.492016 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.491993    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:41:02.658120 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.658034    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") pod \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") "
Oct 30 22:41:02.658301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.658121    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\") pod \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") "
Oct 30 22:41:02.658301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.658150    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\") pod \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") "
Oct 30 22:41:02.658301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.658190    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\") pod \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\" (UID: \"3d207c0c-4058-4eae-bcfd-3839d6c7ba37\") "
Oct 30 22:41:02.658525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.658502    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca" pod "3d207c0c-4058-4eae-bcfd-3839d6c7ba37" (UID: "3d207c0c-4058-4eae-bcfd-3839d6c7ba37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:02.660468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.660441    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs" pod "3d207c0c-4058-4eae-bcfd-3839d6c7ba37" (UID: "3d207c0c-4058-4eae-bcfd-3839d6c7ba37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:02.660468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.660457    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls" pod "3d207c0c-4058-4eae-bcfd-3839d6c7ba37" (UID: "3d207c0c-4058-4eae-bcfd-3839d6c7ba37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:02.660468 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.660444    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb" pod "3d207c0c-4058-4eae-bcfd-3839d6c7ba37" (UID: "3d207c0c-4058-4eae-bcfd-3839d6c7ba37"). InnerVolumeSpecName "kube-api-access-zlrnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:02.759430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.759398    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:02.759430 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.759431    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-kube-api-access-zlrnb\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:02.759637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.759441    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:02.759637 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.759449    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d207c0c-4058-4eae-bcfd-3839d6c7ba37-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:02.987892 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:02.987837    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing" containerID="cri-o://a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d" gracePeriod=25
Oct 30 22:41:03.268568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.268482    4295 generic.go:356] "Generic (PLEG): container finished" podID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerID="afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2" exitCode=0
Oct 30 22:41:03.268568 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.268531    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"
Oct 30 22:41:03.268795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.268565    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" event={"ID":"3d207c0c-4058-4eae-bcfd-3839d6c7ba37","Type":"ContainerDied","Data":"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"}
Oct 30 22:41:03.268795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.268622    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq" event={"ID":"3d207c0c-4058-4eae-bcfd-3839d6c7ba37","Type":"ContainerDied","Data":"c9aa03f78e6238e7f28c3dd594b3c84151c68c2dd9988e44bfdaef0febb226f3"}
Oct 30 22:41:03.268795 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.268650    4295 scope.go:122] "RemoveContainer" containerID="afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"
Oct 30 22:41:03.281370 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.281336    4295 scope.go:122] "RemoveContainer" containerID="afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"
Oct 30 22:41:03.282215 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:41:03.282181    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2\": container with ID starting with afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2 not found: ID does not exist" containerID="afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"
Oct 30 22:41:03.282341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.282228    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2"} err="failed to get container status \"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2\": rpc error: code = NotFound desc = could not find container \"afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2\": container with ID starting with afe8a1256b32ffb2857eb16ca5bc758b34e6c764bcfdc29f57b41920a5aa9aa2 not found: ID does not exist"
Oct 30 22:41:03.438404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.438367    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:41:03.443984 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.443960    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca3b17-694f686f5-sjnhq"]
Oct 30 22:41:03.493046 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:03.493018    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" path="/var/lib/kubelet/pods/3d207c0c-4058-4eae-bcfd-3839d6c7ba37/volumes"
Oct 30 22:41:07.832513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:07.832475    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:41:17.832097 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:17.832057    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:41:23.231924 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.231900    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:41:23.348425 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.348357    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\" (UniqueName: \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\") pod \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") "
Oct 30 22:41:23.348562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.348467    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\") pod \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\" (UID: \"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a\") "
Oct 30 22:41:23.348874 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.348846    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume" pod "40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" (UID: "40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:23.350459 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.350438    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb" pod "40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" (UID: "40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a"). InnerVolumeSpecName "kube-api-access-b8ftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:23.358541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.358511    4295 generic.go:356] "Generic (PLEG): container finished" podID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerID="a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d" exitCode=0
Oct 30 22:41:23.358642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.358553    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"
Oct 30 22:41:23.358642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.358576    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" event={"ID":"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a","Type":"ContainerDied","Data":"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"}
Oct 30 22:41:23.358642 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.358620    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk" event={"ID":"40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a","Type":"ContainerDied","Data":"3509b9ce4438c5502754873f44b4d631575869eb3ea155d998f33ba64cd573ae"}
Oct 30 22:41:23.358818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.358643    4295 scope.go:122] "RemoveContainer" containerID="a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"
Oct 30 22:41:23.367622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.367600    4295 scope.go:122] "RemoveContainer" containerID="a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"
Oct 30 22:41:23.367921 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:41:23.367893    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d\": container with ID starting with a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d not found: ID does not exist" containerID="a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"
Oct 30 22:41:23.368017 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.367935    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d"} err="failed to get container status \"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d\": rpc error: code = NotFound desc = could not find container \"a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d\": container with ID starting with a8db97df26c1d4dfc2c421e64e3d85db03773c835a8b0b36bf32638315c1c69d not found: ID does not exist"
Oct 30 22:41:23.449232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.449200    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:23.449232 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.449226    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ftb\" (UniqueName: \"kubernetes.io/projected/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a-kube-api-access-b8ftb\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:23.529803 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.529764    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:41:23.535377 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:23.535350    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk"]
Oct 30 22:41:23.607529 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:41:23.607446    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:41:25.492397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:41:25.492364    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" path="/var/lib/kubelet/pods/40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a/volumes"
Oct 30 22:41:53.606863 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:41:53.606827    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:42:23.607855 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:42:23.607815    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:42:53.606632 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:42:53.606595    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:43:23.607478 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:43:23.607394    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:43:53.607787 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:43:53.607750    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:44:08.568084 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.568041    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:44:08.569324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569294    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerName="maas-api"
Oct 30 22:44:08.569324 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569323    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerName="maas-api"
Oct 30 22:44:08.569536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569387    4295 cpu_manager.go:414] "containerMap: removing container" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing"
Oct 30 22:44:08.569536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569396    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing"
Oct 30 22:44:08.569657 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569575    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e82f4c-4d5e-4ef4-87b5-cd34a6e1947a" containerName="payload-processing"
Oct 30 22:44:08.569657 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.569594    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d207c0c-4058-4eae-bcfd-3839d6c7ba37" containerName="maas-api"
Oct 30 22:44:08.579754 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.579622    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.625329 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.625294    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:44:08.661301 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.661261    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ef21ee\""
Oct 30 22:44:08.663840 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.663815    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.663959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.663856    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hnr\" (UniqueName: \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.673473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.673457    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c7b5f1b5-dockercfg-h2jg5\""
Oct 30 22:44:08.764606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.764563    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.764606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.764602    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hnr\" (UniqueName: \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.765325 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.765305    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.828412 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.828340    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hnr\" (UniqueName: \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\") pod \"payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:08.904140 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:08.904111    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:09.275837 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:09.275811    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:44:09.277822 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:44:09.277795    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbcf1e8_e81d_4ff4_9f37_9e4ca901d248.slice/crio-8b2bca57b367cc93143b75d6e7a9bf71e847d18b18ea423ee4a818b5bbdc8432 WatchSource:0}: Error finding container 8b2bca57b367cc93143b75d6e7a9bf71e847d18b18ea423ee4a818b5bbdc8432: Status 404 returned error can't find the container with id 8b2bca57b367cc93143b75d6e7a9bf71e847d18b18ea423ee4a818b5bbdc8432
Oct 30 22:44:10.115463 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:10.115419    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" event={"ID":"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248","Type":"ContainerStarted","Data":"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"}
Oct 30 22:44:10.115900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:10.115488    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" event={"ID":"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248","Type":"ContainerStarted","Data":"8b2bca57b367cc93143b75d6e7a9bf71e847d18b18ea423ee4a818b5bbdc8432"}
Oct 30 22:44:10.115900 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:10.115565    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:10.236237 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:10.236168    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" podStartSLOduration=2.236146314 podStartE2EDuration="2.236146314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:10.2349946 +0000 UTC m=+3252.877588389" watchObservedRunningTime="2026-10-30 22:44:10.236146314 +0000 UTC m=+3252.878740102"
Oct 30 22:44:11.095733 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.095697    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:44:11.099139 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.099121    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.186712 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.186678    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.187112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.186742    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.187112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.186843    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.187112 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.186923    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7r2\" (UniqueName: \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.203976 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.203950    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ef21ee-serving-cert\""
Oct 30 22:44:11.204121 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.203986    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\""
Oct 30 22:44:11.204180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.204121    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:44:11.205734 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.205716    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:44:11.287612 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.287575    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7r2\" (UniqueName: \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.287821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.287709    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.287821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.287773    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.287821 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.287807    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.288013 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:44:11.287930    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-serving-cert: secret "maas-api-e2e-worker-single-main-ef21ee-serving-cert" not found
Oct 30 22:44:11.288069 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:44:11.288019    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls podName:b741ad4e-6cac-445b-9504-4cc19f5d3545 nodeName:}" failed. No retries permitted until 2026-10-30 22:44:11.787995948 +0000 UTC m=+3254.430589732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls") pod "maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" (UID: "b741ad4e-6cac-445b-9504-4cc19f5d3545") : secret "maas-api-e2e-worker-single-main-ef21ee-serving-cert" not found
Oct 30 22:44:11.288373 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.288353    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.290292 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.290265    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.359186 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.359078    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7r2\" (UniqueName: \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.792327 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.792297    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:11.794998 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:11.794971    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:12.012089 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:12.012052    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:12.386009 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:12.385982    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:44:12.387898 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:44:12.387868    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb741ad4e_6cac_445b_9504_4cc19f5d3545.slice/crio-e2e8ecb2c487ee20d2854371a0ba2a33779db12bea2299662ca02d5be982f959 WatchSource:0}: Error finding container e2e8ecb2c487ee20d2854371a0ba2a33779db12bea2299662ca02d5be982f959: Status 404 returned error can't find the container with id e2e8ecb2c487ee20d2854371a0ba2a33779db12bea2299662ca02d5be982f959
Oct 30 22:44:13.129532 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:13.129492    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" event={"ID":"b741ad4e-6cac-445b-9504-4cc19f5d3545","Type":"ContainerStarted","Data":"e2e8ecb2c487ee20d2854371a0ba2a33779db12bea2299662ca02d5be982f959"}
Oct 30 22:44:14.135517 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:14.135476    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" event={"ID":"b741ad4e-6cac-445b-9504-4cc19f5d3545","Type":"ContainerStarted","Data":"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"}
Oct 30 22:44:14.135982 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:14.135561    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:14.292039 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:14.291977    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" podStartSLOduration=2.361701405 podStartE2EDuration="3.291961859s" totalImagesPullingTime="930.260454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:14.290901427 +0000 UTC m=+3256.933495218" watchObservedRunningTime="2026-10-30 22:44:14.291961859 +0000 UTC m=+3256.934555648"
Oct 30 22:44:20.145216 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:20.145187    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:44:21.122476 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:44:21.122444    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:44:23.607452 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:44:23.607415    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:44:53.607202 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:44:53.607164    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:45:23.608086 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:45:23.608050    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:45:40.736469 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:40.736433    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:45:43.264747 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:43.264711    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:45:43.265163 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:43.264896    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" podUID="b741ad4e-6cac-445b-9504-4cc19f5d3545" containerName="maas-api" containerID="cri-o://37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a" gracePeriod=30
Oct 30 22:45:44.511513 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.511483    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:45:44.513791 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.513765    4295 generic.go:356] "Generic (PLEG): container finished" podID="b741ad4e-6cac-445b-9504-4cc19f5d3545" containerID="37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a" exitCode=0
Oct 30 22:45:44.513910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.513792    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"
Oct 30 22:45:44.513910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.513842    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" event={"ID":"b741ad4e-6cac-445b-9504-4cc19f5d3545","Type":"ContainerDied","Data":"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"}
Oct 30 22:45:44.513910 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.513888    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c" event={"ID":"b741ad4e-6cac-445b-9504-4cc19f5d3545","Type":"ContainerDied","Data":"e2e8ecb2c487ee20d2854371a0ba2a33779db12bea2299662ca02d5be982f959"}
Oct 30 22:45:44.514026 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.513911    4295 scope.go:122] "RemoveContainer" containerID="37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"
Oct 30 22:45:44.522947 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.522928    4295 scope.go:122] "RemoveContainer" containerID="37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"
Oct 30 22:45:44.523208 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:45:44.523189    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a\": container with ID starting with 37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a not found: ID does not exist" containerID="37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"
Oct 30 22:45:44.523259 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.523215    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a"} err="failed to get container status \"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a\": rpc error: code = NotFound desc = could not find container \"37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a\": container with ID starting with 37fc6e4c79249bd34ea63de33d7d68f7628bb10777395e268a0d30f9b4e60e5a not found: ID does not exist"
Oct 30 22:45:44.561529 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.561501    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\" (UniqueName: \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\") pod \"b741ad4e-6cac-445b-9504-4cc19f5d3545\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") "
Oct 30 22:45:44.561648 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.561581    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\") pod \"b741ad4e-6cac-445b-9504-4cc19f5d3545\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") "
Oct 30 22:45:44.561720 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.561657    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\") pod \"b741ad4e-6cac-445b-9504-4cc19f5d3545\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") "
Oct 30 22:45:44.561783 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.561767    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") pod \"b741ad4e-6cac-445b-9504-4cc19f5d3545\" (UID: \"b741ad4e-6cac-445b-9504-4cc19f5d3545\") "
Oct 30 22:45:44.562039 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.562006    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca" pod "b741ad4e-6cac-445b-9504-4cc19f5d3545" (UID: "b741ad4e-6cac-445b-9504-4cc19f5d3545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:44.562171 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.562132    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741ad4e-6cac-445b-9504-4cc19f5d3545-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:44.563887 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.563835    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2" pod "b741ad4e-6cac-445b-9504-4cc19f5d3545" (UID: "b741ad4e-6cac-445b-9504-4cc19f5d3545"). InnerVolumeSpecName "kube-api-access-ps7r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:44.563957 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.563926    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls" pod "b741ad4e-6cac-445b-9504-4cc19f5d3545" (UID: "b741ad4e-6cac-445b-9504-4cc19f5d3545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:44.564004 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.563969    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs" pod "b741ad4e-6cac-445b-9504-4cc19f5d3545" (UID: "b741ad4e-6cac-445b-9504-4cc19f5d3545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:44.663262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.663224    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:44.663262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.663254    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741ad4e-6cac-445b-9504-4cc19f5d3545-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:44.663262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.663268    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7r2\" (UniqueName: \"kubernetes.io/projected/b741ad4e-6cac-445b-9504-4cc19f5d3545-kube-api-access-ps7r2\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:44.937364 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.937332    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:45:44.943167 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:44.943137    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ef21ee-648bb869bb-x2b2c"]
Oct 30 22:45:45.495250 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:45.495209    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b741ad4e-6cac-445b-9504-4cc19f5d3545" path="/var/lib/kubelet/pods/b741ad4e-6cac-445b-9504-4cc19f5d3545/volumes"
Oct 30 22:45:45.737534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:45.737470    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing" containerID="cri-o://76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40" gracePeriod=25
Oct 30 22:45:51.121622 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:45:51.121580    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:45:53.607081 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:45:53.607045    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:46:01.120938 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:01.120900    4295 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:05.986525 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:05.986498    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:46:06.160149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.160073    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\") pod \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") "
Oct 30 22:46:06.160149 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.160135    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\" (UniqueName: \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\") pod \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\" (UID: \"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248\") "
Oct 30 22:46:06.160447 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.160424    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume" pod "2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" (UID: "2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:06.162268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.162248    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr" pod "2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" (UID: "2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248"). InnerVolumeSpecName "kube-api-access-j5hnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:06.261716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.261688    4295 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-plugins-config-volume\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:46:06.261716 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.261714    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hnr\" (UniqueName: \"kubernetes.io/projected/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248-kube-api-access-j5hnr\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:46:06.608426 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.608391    4295 generic.go:356] "Generic (PLEG): container finished" podID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerID="76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40" exitCode=0
Oct 30 22:46:06.608595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.608437    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"
Oct 30 22:46:06.608595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.608439    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" event={"ID":"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248","Type":"ContainerDied","Data":"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"}
Oct 30 22:46:06.608595 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.608566    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc" event={"ID":"2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248","Type":"ContainerDied","Data":"8b2bca57b367cc93143b75d6e7a9bf71e847d18b18ea423ee4a818b5bbdc8432"}
Oct 30 22:46:06.608748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.608603    4295 scope.go:122] "RemoveContainer" containerID="76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"
Oct 30 22:46:06.616963 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.616947    4295 scope.go:122] "RemoveContainer" containerID="76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"
Oct 30 22:46:06.617256 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:46:06.617239    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40\": container with ID starting with 76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40 not found: ID does not exist" containerID="76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"
Oct 30 22:46:06.617304 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.617264    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40"} err="failed to get container status \"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40\": rpc error: code = NotFound desc = could not find container \"76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40\": container with ID starting with 76547db85447f6039a30f91a7ae1006a7593cc9cdacd783160bedee5e3308a40 not found: ID does not exist"
Oct 30 22:46:06.779700 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.779651    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:46:06.830348 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:06.830320    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc"]
Oct 30 22:46:07.495477 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:46:07.495111    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" path="/var/lib/kubelet/pods/2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248/volumes"
Oct 30 22:46:23.607182 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:46:23.607094    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:46:53.685975 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:46:53.685940    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:47:17.030880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.030843    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"]
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.031911    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b741ad4e-6cac-445b-9504-4cc19f5d3545" containerName="maas-api"
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.031940    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b741ad4e-6cac-445b-9504-4cc19f5d3545" containerName="maas-api"
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.032011    4295 cpu_manager.go:414] "containerMap: removing container" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing"
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.032019    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing"
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.032207    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b741ad4e-6cac-445b-9504-4cc19f5d3545" containerName="maas-api"
Oct 30 22:47:17.033863 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.032232    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbcf1e8-e81d-4ff4-9f37-9e4ca901d248" containerName="payload-processing"
Oct 30 22:47:17.037236 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.036101    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.084452 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.084421    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.084594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.084462    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.084594 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.084551    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkgfx\" (UniqueName: \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.094771 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.094736    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"]
Oct 30 22:47:17.142859 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.142830    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:47:17.142859 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.142834    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:17.185608 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.185576    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.185762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.185621    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.185762 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.185700    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkgfx\" (UniqueName: \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.186381 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.186354    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.188275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.188251    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.249421 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.249389    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkgfx\" (UniqueName: \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\") pod \"maas-api-revoke-sub-452b6d7b-cmlcc\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.350375 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.350292    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:17.722959 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.722934    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"]
Oct 30 22:47:17.725581 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:47:17.725555    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b629caa_79ec_4eaf_9af2_d9c061847552.slice/crio-d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8 WatchSource:0}: Error finding container d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8: Status 404 returned error can't find the container with id d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8
Oct 30 22:47:17.975186 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:17.975099    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc" event={"ID":"6b629caa-79ec-4eaf-9af2-d9c061847552","Type":"ContainerStarted","Data":"d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8"}
Oct 30 22:47:18.981480 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:18.981445    4295 generic.go:356] "Generic (PLEG): container finished" podID="6b629caa-79ec-4eaf-9af2-d9c061847552" containerID="41d3083b092cee300b95a1b2de7ee34880fbe0d2efb81c367583622e6089efd5" exitCode=0
Oct 30 22:47:18.982057 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:18.981514    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc" event={"ID":"6b629caa-79ec-4eaf-9af2-d9c061847552","Type":"ContainerDied","Data":"41d3083b092cee300b95a1b2de7ee34880fbe0d2efb81c367583622e6089efd5"}
Oct 30 22:47:20.722035 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.721916    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:20.770851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.769963    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\") pod \"6b629caa-79ec-4eaf-9af2-d9c061847552\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") "
Oct 30 22:47:20.770851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.770091    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\") pod \"6b629caa-79ec-4eaf-9af2-d9c061847552\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") "
Oct 30 22:47:20.770851 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.770163    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\" (UniqueName: \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\") pod \"6b629caa-79ec-4eaf-9af2-d9c061847552\" (UID: \"6b629caa-79ec-4eaf-9af2-d9c061847552\") "
Oct 30 22:47:20.771763 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.771713    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca" pod "6b629caa-79ec-4eaf-9af2-d9c061847552" (UID: "6b629caa-79ec-4eaf-9af2-d9c061847552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:20.786156 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.786119    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca" pod "6b629caa-79ec-4eaf-9af2-d9c061847552" (UID: "6b629caa-79ec-4eaf-9af2-d9c061847552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:20.790446 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.790406    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx" pod "6b629caa-79ec-4eaf-9af2-d9c061847552" (UID: "6b629caa-79ec-4eaf-9af2-d9c061847552"). InnerVolumeSpecName "kube-api-access-wkgfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:20.876355 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.876253    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkgfx\" (UniqueName: \"kubernetes.io/projected/6b629caa-79ec-4eaf-9af2-d9c061847552-kube-api-access-wkgfx\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:20.876979 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.876957    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:20.877133 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:20.877111    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b629caa-79ec-4eaf-9af2-d9c061847552-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:21.029703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:21.027378    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc" event={"ID":"6b629caa-79ec-4eaf-9af2-d9c061847552","Type":"ContainerDied","Data":"d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8"}
Oct 30 22:47:21.029703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:21.027444    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e5c27f8f1017c14984b803b1fa87147df90c651927d6eb6afeed2ca4037cc8"
Oct 30 22:47:21.029703 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:21.027518    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"
Oct 30 22:47:23.607786 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:47:23.607748    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:47:42.835957 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.835875    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"]
Oct 30 22:47:42.836470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.836446    4295 cpu_manager.go:414] "containerMap: removing container" podUID="6b629caa-79ec-4eaf-9af2-d9c061847552" containerName="revoke-keys"
Oct 30 22:47:42.836515 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.836476    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b629caa-79ec-4eaf-9af2-d9c061847552" containerName="revoke-keys"
Oct 30 22:47:42.836611 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.836598    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b629caa-79ec-4eaf-9af2-d9c061847552" containerName="revoke-keys"
Oct 30 22:47:42.839907 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.839885    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:42.898587 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.898546    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"]
Oct 30 22:47:42.940854 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.940822    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:42.941044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.940866    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf45g\" (UniqueName: \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:42.941044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.940978    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:42.946470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.946444    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:42.946735 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:42.946720    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:47:43.041702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.041643    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.041871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.041739    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf45g\" (UniqueName: \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.041871 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.041811    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.042507 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.042392    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.044191 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.044133    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.106280 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.106210    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf45g\" (UniqueName: \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\") pod \"maas-api-revoke-sub-3e31a506-r9mpq\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.152293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.152260    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:43.526898 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:43.526869    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"]
Oct 30 22:47:43.530485 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:47:43.530434    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931e6abd_43ec_488e_9af4_ff1b5bb8c95e.slice/crio-3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4 WatchSource:0}: Error finding container 3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4: Status 404 returned error can't find the container with id 3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4
Oct 30 22:47:44.207418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:44.207302    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq" event={"ID":"931e6abd-43ec-488e-9af4-ff1b5bb8c95e","Type":"ContainerStarted","Data":"566f8d017ae9ce3d56b78b7a816115c71f0701610e3ea2f6e55e205d4b52fd71"}
Oct 30 22:47:44.207418 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:44.207376    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq" event={"ID":"931e6abd-43ec-488e-9af4-ff1b5bb8c95e","Type":"ContainerStarted","Data":"3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4"}
Oct 30 22:47:44.333274 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:44.333208    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq" podStartSLOduration=2.333188639 podStartE2EDuration="2.333188639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:44.331923064 +0000 UTC m=+3466.974516857" watchObservedRunningTime="2026-10-30 22:47:44.333188639 +0000 UTC m=+3466.975782431"
Oct 30 22:47:45.238339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:45.238299    4295 generic.go:356] "Generic (PLEG): container finished" podID="931e6abd-43ec-488e-9af4-ff1b5bb8c95e" containerID="566f8d017ae9ce3d56b78b7a816115c71f0701610e3ea2f6e55e205d4b52fd71" exitCode=0
Oct 30 22:47:45.239486 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:45.239403    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq" event={"ID":"931e6abd-43ec-488e-9af4-ff1b5bb8c95e","Type":"ContainerDied","Data":"566f8d017ae9ce3d56b78b7a816115c71f0701610e3ea2f6e55e205d4b52fd71"}
Oct 30 22:47:46.377573 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.377545    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:46.467361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.467331    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\" (UniqueName: \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\") pod \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") "
Oct 30 22:47:46.467562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.467415    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\") pod \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") "
Oct 30 22:47:46.467562 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.467480    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\") pod \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\" (UID: \"931e6abd-43ec-488e-9af4-ff1b5bb8c95e\") "
Oct 30 22:47:46.467888 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.467863    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca" pod "931e6abd-43ec-488e-9af4-ff1b5bb8c95e" (UID: "931e6abd-43ec-488e-9af4-ff1b5bb8c95e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.469656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.469634    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca" pod "931e6abd-43ec-488e-9af4-ff1b5bb8c95e" (UID: "931e6abd-43ec-488e-9af4-ff1b5bb8c95e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:46.469656 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.469685    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g" pod "931e6abd-43ec-488e-9af4-ff1b5bb8c95e" (UID: "931e6abd-43ec-488e-9af4-ff1b5bb8c95e"). InnerVolumeSpecName "kube-api-access-rf45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:46.569096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.569017    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-openshift-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.569096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.569053    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-maas-api-xks-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.569096 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:46.569070    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf45g\" (UniqueName: \"kubernetes.io/projected/931e6abd-43ec-488e-9af4-ff1b5bb8c95e-kube-api-access-rf45g\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.250315 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:47.250283    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"
Oct 30 22:47:47.250470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:47.250282    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq" event={"ID":"931e6abd-43ec-488e-9af4-ff1b5bb8c95e","Type":"ContainerDied","Data":"3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4"}
Oct 30 22:47:47.250470 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:47:47.250413    4295 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af3a91b5ba8492dca22574d2cdf20086aed814947ac98962b39d7921c4b1bf4"
Oct 30 22:47:53.607547 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:47:53.607442    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:48:23.606579 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:48:23.606537    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:48:25.720444 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.720406    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:25.720935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.720909    4295 cpu_manager.go:414] "containerMap: removing container" podUID="931e6abd-43ec-488e-9af4-ff1b5bb8c95e" containerName="revoke-keys"
Oct 30 22:48:25.720935 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.720934    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931e6abd-43ec-488e-9af4-ff1b5bb8c95e" containerName="revoke-keys"
Oct 30 22:48:25.721154 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.721056    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="931e6abd-43ec-488e-9af4-ff1b5bb8c95e" containerName="revoke-keys"
Oct 30 22:48:25.724293 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.724274    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.744275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.744255    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-f42ct"]
Oct 30 22:48:25.747534 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.747519    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.793749 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.793717    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\""
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811337    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811401    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811432    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811590    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-maas-api-tls\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811646    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811698    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-metrics-certs\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811792    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03654af-3e9c-4232-8475-0dafcd637770-openshift-service-ca\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.811926 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.811836    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h25x\" (UniqueName: \"kubernetes.io/projected/b03654af-3e9c-4232-8475-0dafcd637770-kube-api-access-9h25x\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.826307 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.826280    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:48:25.829653 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.829610    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:48:25.831811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.831786    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:25.833287 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.833268    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-f42ct"]
Oct 30 22:48:25.913077 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913048    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913084    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913104    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913133    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-maas-api-tls\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913169    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913183    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-metrics-certs\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913220    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03654af-3e9c-4232-8475-0dafcd637770-openshift-service-ca\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.913263 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.913246    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h25x\" (UniqueName: \"kubernetes.io/projected/b03654af-3e9c-4232-8475-0dafcd637770-kube-api-access-9h25x\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.914087 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.914060    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03654af-3e9c-4232-8475-0dafcd637770-openshift-service-ca\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.914257 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.914232    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.915845 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.915816    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.915954 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.915932    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-metrics-certs\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.916205 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.916189    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.916278 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.916262    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03654af-3e9c-4232-8475-0dafcd637770-maas-api-tls\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:25.979516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.979450    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\") pod \"maas-api-57fcf7d945-8f8zh\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:25.979639 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:25.979579    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h25x\" (UniqueName: \"kubernetes.io/projected/b03654af-3e9c-4232-8475-0dafcd637770-kube-api-access-9h25x\") pod \"maas-api-857794cb89-f42ct\" (UID: \"b03654af-3e9c-4232-8475-0dafcd637770\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:26.038065 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:26.038024    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:26.066012 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:26.065987    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:26.473137 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:26.473111    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:26.474137 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:48:26.474113    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47ef01c_42ff_45b9_92e7_4af841fc5f6a.slice/crio-e75ac056f75f46c12264d1a98111c0f0906bc6baf4f543870c0080a1e6679998 WatchSource:0}: Error finding container e75ac056f75f46c12264d1a98111c0f0906bc6baf4f543870c0080a1e6679998: Status 404 returned error can't find the container with id e75ac056f75f46c12264d1a98111c0f0906bc6baf4f543870c0080a1e6679998
Oct 30 22:48:26.475379 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:26.475363    4295 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:48:26.480381 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:26.480358    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-f42ct"]
Oct 30 22:48:26.507817 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:48:26.507791    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03654af_3e9c_4232_8475_0dafcd637770.slice/crio-832261e25b51019f73b62ac37959ce54c04f22dd97c8041cd0fd6013ab7ae65c WatchSource:0}: Error finding container 832261e25b51019f73b62ac37959ce54c04f22dd97c8041cd0fd6013ab7ae65c: Status 404 returned error can't find the container with id 832261e25b51019f73b62ac37959ce54c04f22dd97c8041cd0fd6013ab7ae65c
Oct 30 22:48:27.471281 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.471248    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct" event={"ID":"b03654af-3e9c-4232-8475-0dafcd637770","Type":"ContainerStarted","Data":"d34fd4b1a0b082c73a78b16825d660a3812ceaff9df0ec9bffffc0a3650d6628"}
Oct 30 22:48:27.471736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.471300    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct" event={"ID":"b03654af-3e9c-4232-8475-0dafcd637770","Type":"ContainerStarted","Data":"832261e25b51019f73b62ac37959ce54c04f22dd97c8041cd0fd6013ab7ae65c"}
Oct 30 22:48:27.471736 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.471369    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:27.472832 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.472812    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" event={"ID":"b47ef01c-42ff-45b9-92e7-4af841fc5f6a","Type":"ContainerStarted","Data":"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"}
Oct 30 22:48:27.472933 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.472854    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" event={"ID":"b47ef01c-42ff-45b9-92e7-4af841fc5f6a","Type":"ContainerStarted","Data":"e75ac056f75f46c12264d1a98111c0f0906bc6baf4f543870c0080a1e6679998"}
Oct 30 22:48:27.472933 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.472875    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:27.620185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.620083    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct" podStartSLOduration=1.8569505080000002 podStartE2EDuration="2.620070265s" totalImagesPullingTime="763.119757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:27.619051837 +0000 UTC m=+3510.261645627" watchObservedRunningTime="2026-10-30 22:48:27.620070265 +0000 UTC m=+3510.262664054"
Oct 30 22:48:27.742504 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:27.742448    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" podStartSLOduration=1.86705481 podStartE2EDuration="2.742433876s" totalImagesPullingTime="875.379066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:27.741513361 +0000 UTC m=+3510.384107150" watchObservedRunningTime="2026-10-30 22:48:27.742433876 +0000 UTC m=+3510.385027689"
Oct 30 22:48:33.492750 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.492720    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:33.493227 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.492790    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-857794cb89-f42ct"
Oct 30 22:48:33.667555 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.667523    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:33.667818 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.667774    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" podUID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" containerName="maas-api" containerID="cri-o://c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074" gracePeriod=30
Oct 30 22:48:33.918482 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.918458    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:33.998948 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.998918    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\") pod \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") "
Oct 30 22:48:33.999115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.999036    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\") pod \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") "
Oct 30 22:48:33.999115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.999069    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\") pod \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") "
Oct 30 22:48:33.999115 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.999097    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\") pod \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\" (UID: \"b47ef01c-42ff-45b9-92e7-4af841fc5f6a\") "
Oct 30 22:48:33.999440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:33.999412    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca" pod "b47ef01c-42ff-45b9-92e7-4af841fc5f6a" (UID: "b47ef01c-42ff-45b9-92e7-4af841fc5f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:34.001116 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.001093    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm" pod "b47ef01c-42ff-45b9-92e7-4af841fc5f6a" (UID: "b47ef01c-42ff-45b9-92e7-4af841fc5f6a"). InnerVolumeSpecName "kube-api-access-9wjjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:34.001204 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.001189    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs" pod "b47ef01c-42ff-45b9-92e7-4af841fc5f6a" (UID: "b47ef01c-42ff-45b9-92e7-4af841fc5f6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:34.001372 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.001350    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls" pod "b47ef01c-42ff-45b9-92e7-4af841fc5f6a" (UID: "b47ef01c-42ff-45b9-92e7-4af841fc5f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:34.100919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.100840    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-kube-api-access-9wjjm\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.100919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.100870    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.100919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.100880    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.100919 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.100889    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ef01c-42ff-45b9-92e7-4af841fc5f6a-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.511610 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.511572    4295 generic.go:356] "Generic (PLEG): container finished" podID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" containerID="c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074" exitCode=0
Oct 30 22:48:34.512110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.511619    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"
Oct 30 22:48:34.512110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.511674    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" event={"ID":"b47ef01c-42ff-45b9-92e7-4af841fc5f6a","Type":"ContainerDied","Data":"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"}
Oct 30 22:48:34.512110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.511738    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh" event={"ID":"b47ef01c-42ff-45b9-92e7-4af841fc5f6a","Type":"ContainerDied","Data":"e75ac056f75f46c12264d1a98111c0f0906bc6baf4f543870c0080a1e6679998"}
Oct 30 22:48:34.512110 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.511772    4295 scope.go:122] "RemoveContainer" containerID="c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"
Oct 30 22:48:34.520536 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.520520    4295 scope.go:122] "RemoveContainer" containerID="c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"
Oct 30 22:48:34.520814 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:48:34.520796    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074\": container with ID starting with c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074 not found: ID does not exist" containerID="c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"
Oct 30 22:48:34.520880 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.520821    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074"} err="failed to get container status \"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074\": rpc error: code = NotFound desc = could not find container \"c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074\": container with ID starting with c6e2a04ac28418e3ce2d16e25d733cc49884ed70e8a8b20a71a23e498eb95074 not found: ID does not exist"
Oct 30 22:48:34.683211 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.683175    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:34.689409 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:34.689384    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-8f8zh"]
Oct 30 22:48:35.493108 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:35.493078    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" path="/var/lib/kubelet/pods/b47ef01c-42ff-45b9-92e7-4af841fc5f6a/volumes"
Oct 30 22:48:41.385510 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.385475    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:48:41.385937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.385874    4295 cpu_manager.go:414] "containerMap: removing container" podUID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" containerName="maas-api"
Oct 30 22:48:41.385937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.385887    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" containerName="maas-api"
Oct 30 22:48:41.385937 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.385938    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47ef01c-42ff-45b9-92e7-4af841fc5f6a" containerName="maas-api"
Oct 30 22:48:41.390253 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.390233    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.445916 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.445886    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7e1fe5c2-serving-cert\""
Oct 30 22:48:41.451342 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.451312    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:48:41.485183 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.485146    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.485361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.485209    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.485361 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.485341    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.485481 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.485379    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll579\" (UniqueName: \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.586687 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.586631    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.586869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.586698    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.586869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.586754    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.586869 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:48:41.586781    4295 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-serving-cert: secret "maas-api-e2e-ait-7e1fe5c2-serving-cert" not found
Oct 30 22:48:41.586869 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.586792    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll579\" (UniqueName: \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.586869 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:48:41.586852    4295 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls podName:3ddf1da6-f549-4319-99e0-280ca0e3197a nodeName:}" failed. No retries permitted until 2026-10-30 22:48:42.086831275 +0000 UTC m=+3524.729425055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls") pod "maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" (UID: "3ddf1da6-f549-4319-99e0-280ca0e3197a") : secret "maas-api-e2e-ait-7e1fe5c2-serving-cert" not found
Oct 30 22:48:41.587526 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.587502    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.589195 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.589177    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:41.665113 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:41.665027    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll579\" (UniqueName: \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:42.091846 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:42.091800    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:42.094516 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:42.094485    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") pod \"maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:42.302929 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:42.302893    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:42.682579 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:42.682552    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:48:42.684400 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:48:42.684362    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddf1da6_f549_4319_99e0_280ca0e3197a.slice/crio-3ec585c6131bcba32f3403166ec42c93336205e81a19267447b4e097da5e7a1d WatchSource:0}: Error finding container 3ec585c6131bcba32f3403166ec42c93336205e81a19267447b4e097da5e7a1d: Status 404 returned error can't find the container with id 3ec585c6131bcba32f3403166ec42c93336205e81a19267447b4e097da5e7a1d
Oct 30 22:48:43.559203 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:43.559161    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" event={"ID":"3ddf1da6-f549-4319-99e0-280ca0e3197a","Type":"ContainerStarted","Data":"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"}
Oct 30 22:48:43.559339 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:43.559221    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" event={"ID":"3ddf1da6-f549-4319-99e0-280ca0e3197a","Type":"ContainerStarted","Data":"3ec585c6131bcba32f3403166ec42c93336205e81a19267447b4e097da5e7a1d"}
Oct 30 22:48:43.559397 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:43.559374    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:43.727180 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:43.727122    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" podStartSLOduration=1.917282153 podStartE2EDuration="2.727103932s" totalImagesPullingTime="809.821779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:43.726842194 +0000 UTC m=+3526.369435984" watchObservedRunningTime="2026-10-30 22:48:43.727103932 +0000 UTC m=+3526.369697725"
Oct 30 22:48:49.569069 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:48:49.569037    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:48:53.608428 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:48:53.608390    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:49:23.607339 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:49:23.607300    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:49:33.167308 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.167273    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:49:33.168003 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.167472    4295 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" podUID="3ddf1da6-f549-4319-99e0-280ca0e3197a" containerName="maas-api" containerID="cri-o://65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481" gracePeriod=30
Oct 30 22:49:33.439841 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.439817    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:49:33.596499 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.596468    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") pod \"3ddf1da6-f549-4319-99e0-280ca0e3197a\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") "
Oct 30 22:49:33.596684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.596582    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\" (UniqueName: \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\") pod \"3ddf1da6-f549-4319-99e0-280ca0e3197a\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") "
Oct 30 22:49:33.596684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.596615    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\") pod \"3ddf1da6-f549-4319-99e0-280ca0e3197a\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") "
Oct 30 22:49:33.596684 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.596654    4295 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\") pod \"3ddf1da6-f549-4319-99e0-280ca0e3197a\" (UID: \"3ddf1da6-f549-4319-99e0-280ca0e3197a\") "
Oct 30 22:49:33.597066 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.597034    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca" pod "3ddf1da6-f549-4319-99e0-280ca0e3197a" (UID: "3ddf1da6-f549-4319-99e0-280ca0e3197a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:33.598918 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.598892    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579" pod "3ddf1da6-f549-4319-99e0-280ca0e3197a" (UID: "3ddf1da6-f549-4319-99e0-280ca0e3197a"). InnerVolumeSpecName "kube-api-access-ll579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:33.599076 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.599048    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs" pod "3ddf1da6-f549-4319-99e0-280ca0e3197a" (UID: "3ddf1da6-f549-4319-99e0-280ca0e3197a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:33.599235 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.599217    4295 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls" pod "3ddf1da6-f549-4319-99e0-280ca0e3197a" (UID: "3ddf1da6-f549-4319-99e0-280ca0e3197a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:33.698111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.698073    4295 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-maas-api-tls\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:33.698111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.698104    4295 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll579\" (UniqueName: \"kubernetes.io/projected/3ddf1da6-f549-4319-99e0-280ca0e3197a-kube-api-access-ll579\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:33.698111 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.698117    4295 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ddf1da6-f549-4319-99e0-280ca0e3197a-metrics-certs\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:33.698359 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.698128    4295 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddf1da6-f549-4319-99e0-280ca0e3197a-openshift-service-ca\") on node \"ip-10-0-135-59.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:33.783053 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.783010    4295 generic.go:356] "Generic (PLEG): container finished" podID="3ddf1da6-f549-4319-99e0-280ca0e3197a" containerID="65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481" exitCode=0
Oct 30 22:49:33.783220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.783057    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" event={"ID":"3ddf1da6-f549-4319-99e0-280ca0e3197a","Type":"ContainerDied","Data":"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"}
Oct 30 22:49:33.783220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.783065    4295 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"
Oct 30 22:49:33.783220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.783138    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs" event={"ID":"3ddf1da6-f549-4319-99e0-280ca0e3197a","Type":"ContainerDied","Data":"3ec585c6131bcba32f3403166ec42c93336205e81a19267447b4e097da5e7a1d"}
Oct 30 22:49:33.783220 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.783161    4295 scope.go:122] "RemoveContainer" containerID="65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"
Oct 30 22:49:33.791654 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.791634    4295 scope.go:122] "RemoveContainer" containerID="65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"
Oct 30 22:49:33.791930 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:49:33.791904    4295 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481\": container with ID starting with 65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481 not found: ID does not exist" containerID="65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"
Oct 30 22:49:33.792031 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.791940    4295 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481"} err="failed to get container status \"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481\": rpc error: code = NotFound desc = could not find container \"65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481\": container with ID starting with 65173fc72ce821d5e2ab79929fbc521ee8685bd7b65b2f95eb82879a4903b481 not found: ID does not exist"
Oct 30 22:49:33.953617 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.953531    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:49:33.960196 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:33.960168    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1fe5c2-5fd49bb457-5bhfs"]
Oct 30 22:49:35.492020 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:49:35.491987    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ddf1da6-f549-4319-99e0-280ca0e3197a" path="/var/lib/kubelet/pods/3ddf1da6-f549-4319-99e0-280ca0e3197a/volumes"
Oct 30 22:49:53.607771 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:49:53.607732    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:50:23.624294 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:50:23.624257    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:50:53.610908 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:50:53.610871    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:51:23.608385 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:51:23.608346    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:51:53.606837 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:51:53.606786    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:52:21.057603 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:21.057510    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"]
Oct 30 22:52:21.108702 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:21.108644    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cmlcc"]
Oct 30 22:52:21.493606 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:21.493576    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b629caa-79ec-4eaf-9af2-d9c061847552" path="/var/lib/kubelet/pods/6b629caa-79ec-4eaf-9af2-d9c061847552/volumes"
Oct 30 22:52:23.606881 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:52:23.606842    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:52:47.057071 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:47.057035    4295 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"]
Oct 30 22:52:47.107262 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:47.107219    4295 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-r9mpq"]
Oct 30 22:52:47.492748 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:52:47.492716    4295 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931e6abd-43ec-488e-9af4-ff1b5bb8c95e" path="/var/lib/kubelet/pods/931e6abd-43ec-488e-9af4-ff1b5bb8c95e/volumes"
Oct 30 22:52:53.607251 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:52:53.607212    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:53:21.611816 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:21.611784    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:53:22.814158 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:22.814129    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5ncvs_708793d9-9768-4314-8207-383e31bfb542/proxy/0.log"
Oct 30 22:53:23.608165 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:53:23.608121    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:53:23.877314 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:23.877235    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857794cb89-f42ct_b03654af-3e9c-4232-8475-0dafcd637770/maas-api/0.log"
Oct 30 22:53:24.859404 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:24.859374    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-54lpm_a903a436-87b2-4e71-b7b8-0f21affb8e70/postgres/0.log"
Oct 30 22:53:29.185817 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:29.185786    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-72njd_aef232b8-14d6-4220-aaba-ba04a64721ea/discovery/0.log"
Oct 30 22:53:29.547875 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:29.547800    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b76569bf4-j5z6p_cc6c1b07-5564-47bd-9009-4ba1d388dec5/kube-auth-proxy/0.log"
Oct 30 22:53:29.987593 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:29.987559    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-2wc6x_f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4/istio-proxy/0.log"
Oct 30 22:53:30.217227 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:30.217195    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-5l25z_7a0403e8-735c-4807-92d8-7a0fed332e42/istio-proxy/0.log"
Oct 30 22:53:31.561798 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:31.561767    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-vmh57_dafe4d11-269b-499c-a118-520bdf759084/istio-proxy/0.log"
Oct 30 22:53:31.891889 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:31.891814    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-whxxj_310492ba-b2e0-4610-90f6-6fb5d5deb753/istio-proxy/0.log"
Oct 30 22:53:34.826680 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:34.826634    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-76862_dc9c6d09-76e0-4fd0-912e-b4f7c9817890/main/0.log"
Oct 30 22:53:36.028711 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:36.028658    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp_ac8e5465-9769-41b1-9526-75740bec3746/main/0.log"
Oct 30 22:53:36.297557 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:36.297484    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn_69d373d5-5760-4d37-8351-8668586c8f88/main/0.log"
Oct 30 22:53:47.649275 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:47.649199    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:53:48.721991 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:48.721959    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5ncvs_708793d9-9768-4314-8207-383e31bfb542/proxy/0.log"
Oct 30 22:53:49.634828 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:49.634795    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857794cb89-f42ct_b03654af-3e9c-4232-8475-0dafcd637770/maas-api/0.log"
Oct 30 22:53:50.601309 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:50.601279    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-54lpm_a903a436-87b2-4e71-b7b8-0f21affb8e70/postgres/0.log"
Oct 30 22:53:53.607033 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:53:53.606987    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:53:54.305707 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:54.305653    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-72njd_aef232b8-14d6-4220-aaba-ba04a64721ea/discovery/0.log"
Oct 30 22:53:54.675872 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:54.675800    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b76569bf4-j5z6p_cc6c1b07-5564-47bd-9009-4ba1d388dec5/kube-auth-proxy/0.log"
Oct 30 22:53:55.127021 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:55.126991    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-2wc6x_f5bf7d3d-a4f2-41d8-a647-08f8a80e87a4/istio-proxy/0.log"
Oct 30 22:53:55.352267 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:55.352238    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-5l25z_7a0403e8-735c-4807-92d8-7a0fed332e42/istio-proxy/0.log"
Oct 30 22:53:56.705563 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:56.705531    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-vmh57_dafe4d11-269b-499c-a118-520bdf759084/istio-proxy/0.log"
Oct 30 22:53:57.032341 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:57.032272    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-whxxj_310492ba-b2e0-4610-90f6-6fb5d5deb753/istio-proxy/0.log"
Oct 30 22:53:58.686615 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:58.686584    4295 scope.go:122] "RemoveContainer" containerID="566f8d017ae9ce3d56b78b7a816115c71f0701610e3ea2f6e55e205d4b52fd71"
Oct 30 22:53:58.694596 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:58.694578    4295 scope.go:122] "RemoveContainer" containerID="41d3083b092cee300b95a1b2de7ee34880fbe0d2efb81c367583622e6089efd5"
Oct 30 22:53:59.475541 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:53:59.475513    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-76862_dc9c6d09-76e0-4fd0-912e-b4f7c9817890/main/0.log"
Oct 30 22:54:00.678185 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:54:00.678152    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zvkzp_ac8e5465-9769-41b1-9526-75740bec3746/main/0.log"
Oct 30 22:54:00.957011 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:54:00.956929    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kzbzn_69d373d5-5760-4d37-8351-8668586c8f88/main/0.log"
Oct 30 22:54:23.606715 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:54:23.606653    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:54:28.436092 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:54:28.436059    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:54:28.956636 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:54:28.956609    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:54:53.607220 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:54:53.607184    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:55:05.670835 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:05.670800    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:55:06.241241 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:06.241204    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb4f557f9-t8dxn_d0a4caa9-4234-4a62-9e4f-4bf9c1c8571f/manager/0.log"
Oct 30 22:55:17.117182 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:17.117108    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jhchm_8b478c4f-7211-4b12-ab86-cd866de1ae8a/global-pull-secret-syncer/0.log"
Oct 30 22:55:21.418244 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:21.418211    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nj82c_72f8a528-1c9b-4f51-8e8f-f298e8982c98/konnectivity-agent/0.log"
Oct 30 22:55:21.965459 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:21.965427    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-59.us-west-1.compute.internal_106668d41ee4901fe4ed1e1a301dc3c3/haproxy/0.log"
Oct 30 22:55:23.608010 ip-10-0-135-59 kubenswrapper[4295]: E1030 22:55:23.607975    4295 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5hv" containerName="extractor"
Oct 30 22:55:27.259166 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.259131    4295 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"]
Oct 30 22:55:27.259567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.259477    4295 cpu_manager.go:414] "containerMap: removing container" podUID="3ddf1da6-f549-4319-99e0-280ca0e3197a" containerName="maas-api"
Oct 30 22:55:27.259567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.259489    4295 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddf1da6-f549-4319-99e0-280ca0e3197a" containerName="maas-api"
Oct 30 22:55:27.259567 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.259548    4295 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddf1da6-f549-4319-99e0-280ca0e3197a" containerName="maas-api"
Oct 30 22:55:27.262537 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.262516    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.317218 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.317183    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"openshift-service-ca.crt\""
Oct 30 22:55:27.317413 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.317244    4295 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"kube-root-ca.crt\""
Oct 30 22:55:27.359268 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.359218    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9fm\" (UniqueName: \"kubernetes.io/projected/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-kube-api-access-rn9fm\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.359440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.359296    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-podres\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.359440 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.359412    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-proc\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.359569 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.359457    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-lib-modules\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.359569 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.359484    4295 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-sys\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.367811 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.367785    4295 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8jvzx\"/\"default-dockercfg-sdt8t\""
Oct 30 22:55:27.369570 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.369548    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"]
Oct 30 22:55:27.460044 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460009    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-podres\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460054    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-proc\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460075    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-lib-modules\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460100    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-sys\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460160    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-proc\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460161    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-podres\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460178    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-lib-modules\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460206    4295 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9fm\" (UniqueName: \"kubernetes.io/projected/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-kube-api-access-rn9fm\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.460240 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.460223    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-sys\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.524249 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.524176    4295 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9fm\" (UniqueName: \"kubernetes.io/projected/08afdf11-7be3-46b6-a7c8-6a5d8ea3492b-kube-api-access-rn9fm\") pod \"perf-node-gather-daemonset-2njsl\" (UID: \"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.575403 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.575367    4295 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:27.946473 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:27.946447    4295 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"]
Oct 30 22:55:27.948706 ip-10-0-135-59 kubenswrapper[4295]: W1030 22:55:27.948676    4295 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod08afdf11_7be3_46b6_a7c8_6a5d8ea3492b.slice/crio-d1f7536e5632faecb8751090af10640dd2b8588c5b36c1cf0e4c9bf6c9a60868 WatchSource:0}: Error finding container d1f7536e5632faecb8751090af10640dd2b8588c5b36c1cf0e4c9bf6c9a60868: Status 404 returned error can't find the container with id d1f7536e5632faecb8751090af10640dd2b8588c5b36c1cf0e4c9bf6c9a60868
Oct 30 22:55:28.463188 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:28.463156    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl" event={"ID":"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b","Type":"ContainerStarted","Data":"28c1912575bcf1244b350c351a653bfc7fdece26b054bcb87132d27f61c4a142"}
Oct 30 22:55:28.463614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:28.463216    4295 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl" event={"ID":"08afdf11-7be3-46b6-a7c8-6a5d8ea3492b","Type":"ContainerStarted","Data":"d1f7536e5632faecb8751090af10640dd2b8588c5b36c1cf0e4c9bf6c9a60868"}
Oct 30 22:55:28.463614 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:28.463246    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:28.587029 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:28.586977    4295 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl" podStartSLOduration=1.586963281 podStartE2EDuration="1.586963281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:28.585695913 +0000 UTC m=+3931.228289704" watchObservedRunningTime="2026-10-30 22:55:28.586963281 +0000 UTC m=+3931.229557064"
Oct 30 22:55:34.477006 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:34.476972    4295 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-2njsl"
Oct 30 22:55:37.803349 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:37.803324    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rb6f_ab8ebc85-ba38-41f1-a38e-80bdcffc6005/node-exporter/0.log"
Oct 30 22:55:38.230272 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:38.230239    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rb6f_ab8ebc85-ba38-41f1-a38e-80bdcffc6005/kube-rbac-proxy/0.log"
Oct 30 22:55:38.660436 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:38.660366    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rb6f_ab8ebc85-ba38-41f1-a38e-80bdcffc6005/init-textfile/0.log"
Oct 30 22:55:46.002679 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:46.002633    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7mhkb_c9c1409d-00e4-4c0b-8115-c095a39b58fe/prometheus-operator/0.log"
Oct 30 22:55:46.425780 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:46.425691    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7mhkb_c9c1409d-00e4-4c0b-8115-c095a39b58fe/kube-rbac-proxy/0.log"
Oct 30 22:55:46.859459 ip-10-0-135-59 kubenswrapper[4295]: I1030 22:55:46.859431    4295 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7nsmt_8e767adc-6d0b-4e96-8b5b-9c5384fc5a07/prometheus-operator-admission-webhook/0.log"